Tag - amiga

1
Visualizing amiga track data with matlab
2
status updates
3
Amiga Forever
4
on floppy adpll, this time, my solution
5
on recent disk reading results
6
more real world tests
7
found bug last night
8
re-examining XOR data checksum used on amiga floppies
9
working FPGA version of the amiga floppy project
10
Histogram of one track

Visualizing amiga track data with matlab

So looking at a lot of raw data is pretty tough, but matlab handles it with ease.

So the above image shows a sample amiga track from a brand new disk, just recently formatted, within the amiga.  It is basically as perfect of a sample as one could expect.  A few things to notice about it:

  • nice tight groupings
  • Lots of space between ranges
  • No real data points fall in the middle
  • 3 separate ranges all within a reasonable definable tolerance

now let’s look at a bad track.  Now this track was written on my amiga 15-20 years ago, and to tell you how bad this disk was getting —- it literally self-destructed a day or two after I sampled this data.  I’m not sure if I posted about this, but at least one track was completely scraped off the disk —- it could have been lots of wear/tear with the read head constantly touching one particular track, but in any event, the disk was in bad shape.

Now the two images aren’t exactly the same size, so don’t inadvertently read into that.  But now let’s notice what’s so wrong with this picture:

  • much fatter groupings.
  • a bunch of data in no-mans land. Compare especially the space between 4us and 5us.
  • Almost the anti-thesis of the first picture!

You can see where reading the second disk poses some real problems, and I think that some PLL just isn’t going to deal well with this!

Now that this disk is gone, I really can’t work with it.  I’ve got to find other samples that aren’t as bad that I can work with to further refine my hardware/software.

If anyone is interested, I took a capture using my Saleae Logic analyzer and then processed it with a small C program that spits out basically .csv.  Then I imported the .csv into matlab, and turned off lines connecting the data points, and turned on a dot to represent a data point.  And just for clarification, if you haven’t been following the blog, the y index represents time between low-going edges.  The x index is basically the number of edges, so the zero-ith first delta t is on the left, and the rightmost dot would be the last delta t sampled.

I’m very happy to have the ability to visualize this stuff!

status updates

I’ve been more active on the project than the posts have illustrated. I’ve been working on a few different things :

Been trying different PLL solutions, examining how Paula does it, and actually analyzing my read data with Matlab to get a graphical view of the “problem.”  Most of my solutions, while they technically work (ie they have the desired behavior), they don’t actually solve the problem at hand.  I have to get a better hand on what “bad data” looks like, and then try to invent solutions that address the specific problems I’ve seen. I’m also using Paula as one metric of “goodness.”  In my mind, my controller should be better — should read disks that the amiga is not able to.

I don’t think I fully understand the problem.  I know what the SYMPTOMS are — flux transitions are read sometimes as being “in the middle.”, 5us between pulses where 4us=’10’ and 6us=’100′. What the heck does 5us mean?  How do we bias this towards the right direction?  Many of the controllers use some sort of PLL — but I see values one after another that span the acceptable range of values.  You can’t track something that doesn’t trend!

I also want to get a better handle on how Paula works.  I’ve got it hooked back up to the logic analyzer, and been trading messages on English Amiga Board about disk DMA and the like.  I’d like to do automatic comparisons between the output of Paula and my project and see when Paula gets it right, but I get it wrong!

Amiga Forever

I should have mentioned this before, but the Amiga Forever CDs and DVDs are really pretty awesome. The original commodora-era videos, pre-installed games/environment, everything is done very nicely.

I have the 2008 Premium edition and have meant to post about it for a long time. I’ve got to break it out and check it out again. I don’t think I ever watched all the videos.

on floppy adpll, this time, my solution

This isn’t by any stretch finished, but it does do what I’ve expected it to do.

It responds both to differences in phase (counter is set to 0), and differences in frequency (period is adjusted in the direction of the frequency error)

I did this at 3am last night, so there could be a couple bugs in there.

With all this being said, I’m not entirely sure that PLLs are actually required for good reading of amiga floppy disks. My regular “static interval” solution works about 95%-98% of the time. I’m going to come up with a list of problem disks and see if this solution works better/worse/otherwise.

I’ve used a read data window of 1us, which starts being centered on 2us, and is automatically adjusted as data comes in. This produces windows around 2us, 4us, 6us, 8us. I output the overall error which is the deviation from the center of the window as each pulse is received. I’d like to graph this error, but it doesn’t look like Xilinx’s iSim will export a particular output as CSV or whatever.

module floppy_pll(
    input clk,
    input floppy_data,
     input reset,
    output reg window_open,
    output reg [7:0] data,
    output reg strobe,
     output reg [7:0] error
    );

// window is 1us wide
// starts at .5us before counter rollover
// ends .5us after counter rollover
// ideally, edges should be arriving right when the counter rolls over

reg [7:0] period = 100;
reg [10:0] counter = 0;

reg IN_D1, IN_D2;
wire out_negedge = ~IN_D1 & IN_D2;

always@(posedge clk or posedge out_negedge) begin

    if (reset) period <= 100;
   
    if (clk) begin
   
        counter <= counter + 1;
        if (counter == period) counter <= 0;
   
        if ( (counter > (period-25)) || (counter < 25) ) window_open <= 1;
        else window_open <= 0;
   
    end
   
    if (out_negedge) begin
   
        // if counter == 0 and we see out_negedge be positive, then we are perfectly aligned
        // so we dont need to adjust anything
        if (counter != 0) begin
   
            if (window_open) begin
           
                counter <= 0; // align counter to the phase of the incoming signal
               
                if (counter < 25) begin
               
                    //we rolled over before pulse was seen, so make period larger
                    //error values will be over 128
                    period <= period + 1;
                    error <= (128 + counter);
                   
                end
               
                if (counter > (period-25)) begin
               
                    //we haven't rolled when we saw the pulse, so make period shorter
                    //error values will be less than 128
                   
                    period <= period - 1;
                    error <= 128 - (period-counter);
                   
                end
           
            end

        end
       
    end
   
   
    // edge detection flops for data in direct from floppy
    IN_D1 <= floppy_data;
    IN_D2 <= IN_D1;
end

endmodule

on recent disk reading results

(this was posted to Classic Computer mailing list, please disregard if you’re on the list.  I think this is an important topic)

The last two nights I’ve been busy archiving some of my Amiga floppy collection.  Most disks were written over 20 years ago.

On a sample size of about 150 floppies, most of them were perfectly readable by my homegrown usb external amiga floppy drive controller.

I paid very close attention to the failures or ones where my controller struggled.

Without sounding too obvious here, the time between the pulses (which more or less define the data) were grossly out of spec.  The DD pulses should nominally be 4us, 6us, and 8us apart before pre-write compensation.  Most good disks are slightly faster, and normal times for these ranges are:

4us: 3.2-4.2us.  Many around 3.75us
6us: 5.5-6.2us.
8us: 7.5-8.2us

(notice margins around 1-1.3us)

My original microcontroller implementation was 3.2-4.2, 5.2-6.2, and 7.2-8.2.

When my current FPGA controller would have a problem, I’d notice that there were problems right on a boundary.  So maybe pulses were coming in at 3.1us apart instead of 3.2.  Or maybe 4.3 instead of 4.2.  So I kept bumping the intervals apart, making a larger range of pulse times acceptable — the XOR sector checksums were passing, so I was likely making the right choices.  The bits were ending up in the right buckets.

But as I went through some of these disks, I ended up with the difference between ranges(and basically my noise margin) being reduced smaller and smaller.  Some to the point where an incoming pulse time might fall darn smack in the middle of the noise margin.  Which bucket does THAT one go into?

My approach has been very successful(easily 95%+), but it makes me wonder about Phil’s DiscFerret dynamic adaptive approach where a sample of the incoming data defines the ranges.

Some disk drives and controllers might be faster or slower than others, and if you create custom ranges for each disk (each track?), perhaps you’ll have better luck.

more real world tests

So tonight, I put my new FPGA implementation of the amiga floppy project to good use.  I read some more of my collection of amiga floppies.

GREAT NEWS.

It’s working like a champ.  As a matter of fact, I selected disks which I could not previously read with my microcontroller solution.  I could read 90% of the previously unreadable ones.  Most of the unsolvable problems were related with HD disks (which, based on my earlier posts, some drives handle better than others)  Note this is just temporary until I try other drivers to read the disks — and try covering the HD hole.

I have better visibility on the PC as to the “problem” delta T’s.  So pulses that are slightly too far apart, just on the boundary of what I consider to be valid, if I adjust my boundary accordingly, and now consider them valid, everything is peachy-keen.  I want to add a real-time option in my java client to allow this to be adjusted on the fly.  See problems? Adjust the slider, and problems go away.  Pretty neat.

I didn’t have this visibility when the microcontroller was interpreting the delta T’s.  The microcontroller had no easy feedback method to tell me what was happening.  Having high-level debugging of the information on the PC makes this all possible.

Nice to see the software purring.  There is still plenty of improvements to make, usability to be improved, etc.  But its working very nicely.

found bug last night

With this new FPGA solution, certain tracks would result in what I call a “short read.”  A short read is any received track that contains less than 28,125 delta T’s, aka pulse times.  Given a certain capture time, there are minimum’s and maximum’s of the number of pulses on an amiga track.

If we have a 300 rpm drive, then its 60s/300 = 200ms per revolution.  If the bitcells are 2us wide, then you have at most 200ms/2us = 100,000 bit cells.  The one’s density is at max 50%(raw MFM ’10’), so this means every other cell would contain a 1. So 50,000 pulses, so 50,000 delta T’s.  The minimum one’s density is 25%(raw MFM ‘1000’), so 25,000 pulses.  Now we read more than just one revolution of data, because we will very likely start reading in the middle of a sector.  So instead of 11 sectors worth of read time, we actually need to read 12 sectors worth, to ensure we read the entire sector in which we started.  This is 218.2ms of time minimum.  We could potentially re-assemble data, using some type of circular buffer, but this is more trouble than it’s worth.  I currently read 225ms of data.

225ms / 2us = 56,250 maximum, 28,125 minimum.

I had my FTDI chip, for the usb<->ttl converter, D2XX USB parameters setting the USB transfer size to 57000 bytes.  This is definitely over and above what was needed.  Or so I thought.

I bumped the transfer size from 57000 to 60032 (docs said specifically 64 byte multiples), and everything started working.  I had already narrowed it down that the problem tracks were ones that had a high density, where there were lots and lots of pulses.  So I knew the size of the track was related.  I checked for FIFO overflow, and it wasn’t overflowing.

I’ve got to look when I have a free second, but I think my USB packet size is 4096 bytes.  So 56250+4096 (some amount of padding?) = 60346.   Uh-o, I better bump that to 60,352.  I think the driver (or windows?) that maxes out at 64k transfer size, so I still have a little wiggle room.

Long and short is that it appears to be working much better.  I was glad to find this bug with just a little brainstorming, and getting better visibility into my actual pulses count on the FPGA.

re-examining XOR data checksum used on amiga floppies

So I was wondering exactly how effective the XOR checksum that Commodore used for the amiga sectors was.  If I read a sector, and perform the checksum, and the checksum matches the stored checksum, then the data is the same, right?  Not always.  I had expected it to be better, maybe not MD5 or SHA-1 level, but I expected it to be decent.

I had run into this paper a year ago when I was looking at my transfer checksums.  But this certainly also applies to the amiga sectors, too.

Some good excerpts:

  • the percentage of undetected two-bit
    errors becomes approximately 1/k (k being checksum size), which is rather poor performance for
    a checksum (i.e., 12.5% undetected errors for an 8-bit checksum, and 3.125% for a 32-bit checksum).
  • The XOR checksum has the highest probability
    of undetected errors for all checksum algorithms in this study…..
  • There is generally no reason to continue the common practice of
    using an XOR checksum in new designs, because it has the same software computational cost as an
    addition-based checksum but is only about half as effective at detecting errors.

So I wanted to actually prove myself that XOR is that bad, looking at it from the amiga sector perspective, so I wrote a quick program to:

  1. Create a random block of data 512 bytes long
  2. Calculate the 32-bit XOR checksum based on 32-bit chunks at a time and store it as the last byte (as does the amiga)
  3. Select a number of bit inversions(basically corrupt the data in a specific way) which can affect data and/or stored checksum
  4. Recalculate the checksum of the modified block
  5. If the checksums MATCH, meaning that two different sets of data yield the same checksum, then this is an undetected error.
  6. Count the number of undetected errors vs total errors.

That paper lists the properties of an XOR checksum, and I wanted to compare results:

  • Detects all single-bit errors.  (my testing confirms)
  • Fails to detect some two-bit errors. (my testing confirms, see below)
  • If total number of bit errors is odd, XOR checksum detects it. (confirmed with 1,3,5,7 bit errors)
  • Fails to detect any even number of bit errors that occur in the same bit position of the checksum computational block. (confirmed with 2,4,6,8)

The two-bit errors is really the case that I worry about.  If two bit errors occur in the same bit position, and the inverted oppositely(1—>0 and 0—>1), then it won’t be detected.  So how often does this happen with random data?  Paper’s author, Maxino, says 3.125% of the time.  I can definitely confirm this.  My testing shows 1.8 million hits over 56 million tries.   There are some differences with Amiga data, but I think the results can be the same.  I might load some example amiga sectors and try them.

Once the number of bit errors increase, the probability of them happening in the same bit position, in the same direction, and occurring evenly on each position goes down —- and therefore the overall chance of undetection decreases as well.

4 bit undetected errors happen 0.3% of the time.

6 bit undetected errors happen 0.04% of the time.

8 bit undetected errors happen 0.009% of the time.

I have other testing to do, including running burst error tests.

So for each sector, you really don’t want to see only two bit errors occur.  A single bit error, sure.  Multiple bit errors, ok, because we can detect them.  You don’t want to think you have a good copy when it fact the checksum passing was a false positive.


working FPGA version of the amiga floppy project

So, I’ve been working on the FPGA version of the amiga floppy project for some time.  I just recently had a fair bit of free time, and so everything came together rather quickly!

I’m now able to read amiga floppy disks in using the same Java client software I had developed for use with the Parallax SX microcontroller board.  There were a few minor changes in the software — most notably the read data routine from the hardware.

I’ve written the code in Verilog on a Xilinx Spartan-3e evaluation board.

The various hardware parts I’ve described:

  • UART: Written from scratch, a transmitter and a receiver.   Simple to use, variable baud rates.
  • FIFO: Generated from Xilinx’s CoreGen. This connects the floppy interface to the USB interface. 32k bytes
  • FSM to always empty the FIFO to the PC.  Once something goes in the FIFO, it immediately gets sent to the PC
  • Read-floppy-FSM: Stores 225ms of Delta T’s (aka time between edges) as 8-bit integers into the FIFO.
  • Command FSM: Receives single-character commands from the java software to execute (R for read, U for upper head, L for lower head, etc)
  • Transmit test pattern routine: Sends 32k of data to the PC to test for reliable communication

A couple advantages with the FPGA solution:

  • We transmit the data to the PC as soon as it’s available.  I want to characterize the actual latency, but it should be pretty small.  This is different from my load->FRAM, and then FRAM-> PC method.  This method should be much faster and we’re just not idling for 225ms.
  • Instead of transmitting the bit-sync’d raw MFM to the PC, I’m sending the delta T’s.  While this requires a little more processing on PC, the PC can more easily determine why a particularly sector can’t be read.  For instance, is the time between pulses too small? Too large?  On a fringe edge?  Plus, since the Java decodes these, I can now add sliders for “acceptable delta T’s” for each 4, 6, or 8us windows.  Before that would require modifying the firmware on the microcontroller.  I can also start to do statistical analysis on the pulse times.

I am currently doing about 430ms per track.  This sucks.  I was beating that by 100ms with my microcontroller.  However, the problem is that because a variable amount of data is going to the PC, the PC receiving code does not know when exactly to stop receiving, so there’s a wait-timer which I have to optimize.  Once I receive the minimum amount of data, I wait 100ms since the last received data, and then exit.  I’ve got to put my logic analyzers in place and figure out how to optimize it.

Denis@h3q can read a disk in 43s, which is pretty darn good.  He is using tokens like I mentioned here and here and here.  I’m transferring much more data though, which gives me more information.  I like his time, and maybe that would be a nice goal to beat!  Wow. That’s only 269ms/track.  Hrrrmm…. that’s pretty good.

Histogram of one track

Here’s a histogram of one track off an amiga floppy disk.  Notice the peaks near 4us, 6us, and 8us.

histo788

I’ve got this data from my logic analyzer, crunched through my little C proggie, and then graphed with Excel.

This shows roughly 45,000 delta Ts, the time between each transition.  The first group, of course, the ’10’s, second ‘100’s and the third, ‘1000’s.