Tag - disk

1
Visualizing amiga track data with matlab
2
status updates
3
on floppy adpll, this time, my solution
4
results from adding the PJL
5
on recent disk reading results
6
more real world tests
7
characterizing speed performance of floppy drive controller
8
working FPGA version of the amiga floppy project
9
Histogram of one track
10
got build environment working again, fixed uart code

Visualizing amiga track data with matlab

So looking at a lot of raw data is pretty tough, but matlab handles it with ease.

So the above image shows a sample amiga track from a brand new disk, just recently formatted, within the amiga.  It is basically as perfect of a sample as one could expect.  A few things to notice about it:

  • nice tight groupings
  • Lots of space between ranges
  • No real data points fall in the middle
  • 3 separate ranges all within a reasonable definable tolerance

now let’s look at a bad track.  Now this track was written on my amiga 15-20 years ago, and to tell you how bad this disk was getting —- it literally self-destructed a day or two after I sampled this data.  I’m not sure if I posted about this, but at least one track was completely scraped off the disk —- it could have been lots of wear/tear with the read head constantly touching one particular track, but in any event, the disk was in bad shape.

Now the two images aren’t exactly the same size, so don’t inadvertently read into that.  But now let’s notice what’s so wrong with this picture:

  • much fatter groupings.
  • a bunch of data in no-mans land. Compare especially the space between 4us and 5us.
  • Almost the anti-thesis of the first picture!

You can see where reading the second disk poses some real problems, and I think that some PLL just isn’t going to deal well with this!

Now that this disk is gone, I really can’t work with it.  I’ve got to find other samples that aren’t as bad that I can work with to further refine my hardware/software.

If anyone is interested, I took a capture using my Saleae Logic analyzer and then processed it with a small C program that spits out basically .csv.  Then I imported the .csv into matlab, and turned off lines connecting the data points, and turned on a dot to represent a data point.  And just for clarification, if you haven’t been following the blog, the y index represents time between low-going edges.  The x index is basically the number of edges, so the zero-ith first delta t is on the left, and the rightmost dot would be the last delta t sampled.

I’m very happy to have the ability to visualize this stuff!

status updates

I’ve been more active on the project than the posts have illustrated. I’ve been working on a few different things :

Been trying different PLL solutions, examining how Paula does it, and actually analyzing my read data with Matlab to get a graphical view of the “problem.”  Most of my solutions, while they technically work (ie they have the desired behavior), they don’t actually solve the problem at hand.  I have to get a better hand on what “bad data” looks like, and then try to invent solutions that address the specific problems I’ve seen. I’m also using Paula as one metric of “goodness.”  In my mind, my controller should be better — should read disks that the amiga is not able to.

I don’t think I fully understand the problem.  I know what the SYMPTOMS are — flux transitions are read sometimes as being “in the middle.”, 5us between pulses where 4us=’10’ and 6us=’100′. What the heck does 5us mean?  How do we bias this towards the right direction?  Many of the controllers use some sort of PLL — but I see values one after another that span the acceptable range of values.  You can’t track something that doesn’t trend!

I also want to get a better handle on how Paula works.  I’ve got it hooked back up to the logic analyzer, and been trading messages on English Amiga Board about disk DMA and the like.  I’d like to do automatic comparisons between the output of Paula and my project and see when Paula gets it right, but I get it wrong!

on floppy adpll, this time, my solution

This isn’t by any stretch finished, but it does do what I’ve expected it to do.

It responds both to differences in phase (counter is set to 0), and differences in frequency (period is adjusted in the direction of the frequency error)

I did this at 3am last night, so there could be a couple bugs in there.

With all this being said, I’m not entirely sure that PLLs are actually required for good reading of amiga floppy disks. My regular “static interval” solution works about 95%-98% of the time. I’m going to come up with a list of problem disks and see if this solution works better/worse/otherwise.

I’ve used a read data window of 1us, which starts being centered on 2us, and is automatically adjusted as data comes in. This produces windows around 2us, 4us, 6us, 8us. I output the overall error which is the deviation from the center of the window as each pulse is received. I’d like to graph this error, but it doesn’t look like Xilinx’s iSim will export a particular output as CSV or whatever.

module floppy_pll(
    input clk,
    input floppy_data,
     input reset,
    output reg window_open,
    output reg [7:0] data,
    output reg strobe,
     output reg [7:0] error
    );

// window is 1us wide
// starts at .5us before counter rollover
// ends .5us after counter rollover
// ideally, edges should be arriving right when the counter rolls over

reg [7:0] period = 100;
reg [10:0] counter = 0;

reg IN_D1, IN_D2;
wire out_negedge = ~IN_D1 & IN_D2;

always@(posedge clk or posedge out_negedge) begin

    if (reset) period <= 100;
   
    if (clk) begin
   
        counter <= counter + 1;
        if (counter == period) counter <= 0;
   
        if ( (counter > (period-25)) || (counter < 25) ) window_open <= 1;
        else window_open <= 0;
   
    end
   
    if (out_negedge) begin
   
        // if counter == 0 and we see out_negedge be positive, then we are perfectly aligned
        // so we dont need to adjust anything
        if (counter != 0) begin
   
            if (window_open) begin
           
                counter <= 0; // align counter to the phase of the incoming signal
               
                if (counter < 25) begin
               
                    //we rolled over before pulse was seen, so make period larger
                    //error values will be over 128
                    period <= period + 1;
                    error <= (128 + counter);
                   
                end
               
                if (counter > (period-25)) begin
               
                    //we haven't rolled when we saw the pulse, so make period shorter
                    //error values will be less than 128
                   
                    period <= period - 1;
                    error <= 128 - (period-counter);
                   
                end
           
            end

        end
       
    end
   
   
    // edge detection flops for data in direct from floppy
    IN_D1 <= floppy_data;
    IN_D2 <= IN_D1;
end

endmodule

results from adding the PJL

So, as I mentioned in another post, I added Jim Thompson’s phase-jerked-loop to the project.  Phil had good luck using it with his DiscFerret, but it really didn’t help me.  My problem disks are still problem disks.  And the data returned as a result still have the same problem.

This PJL really didn’t help me.

http://www.analog-innovations.com/SED/FloppyDataExtractor.pdf

I expected the numbers to look a little more stable —- as if we are tracking them better, but I really just didn’t see this.

I’m planning on reverting my changes soon unless some more playing around will help.

on recent disk reading results

(this was posted to Classic Computer mailing list, please disregard if you’re on the list.  I think this is an important topic)

The last two nights I’ve been busy archiving some of my Amiga floppy collection.  Most disks were written over 20 years ago.

On a sample size of about 150 floppies, most of them were perfectly readable by my homegrown usb external amiga floppy drive controller.

I paid very close attention to the failures or ones where my controller struggled.

Without sounding too obvious here, the time between the pulses (which more or less define the data) were grossly out of spec.  The DD pulses should nominally be 4us, 6us, and 8us apart before pre-write compensation.  Most good disks are slightly faster, and normal times for these ranges are:

4us: 3.2-4.2us.  Many around 3.75us
6us: 5.5-6.2us.
8us: 7.5-8.2us

(notice margins around 1-1.3us)

My original microcontroller implementation was 3.2-4.2, 5.2-6.2, and 7.2-8.2.

When my current FPGA controller would have a problem, I’d notice that there were problems right on a boundary.  So maybe pulses were coming in at 3.1us apart instead of 3.2.  Or maybe 4.3 instead of 4.2.  So I kept bumping the intervals apart, making a larger range of pulse times acceptable — the XOR sector checksums were passing, so I was likely making the right choices.  The bits were ending up in the right buckets.

But as I went through some of these disks, I ended up with the difference between ranges(and basically my noise margin) being reduced smaller and smaller.  Some to the point where an incoming pulse time might fall darn smack in the middle of the noise margin.  Which bucket does THAT one go into?

My approach has been very successful(easily 95%+), but it makes me wonder about Phil’s DiscFerret dynamic adaptive approach where a sample of the incoming data defines the ranges.

Some disk drives and controllers might be faster or slower than others, and if you create custom ranges for each disk (each track?), perhaps you’ll have better luck.

more real world tests

So tonight, I put my new FPGA implementation of the amiga floppy project to good use.  I read some more of my collection of amiga floppies.

GREAT NEWS.

It’s working like a champ.  As a matter of fact, I selected disks which I could not previously read with my microcontroller solution.  I could read 90% of the previously unreadable ones.  Most of the unsolvable problems were related with HD disks (which, based on my earlier posts, some drives handle better than others)  Note this is just temporary until I try other drivers to read the disks — and try covering the HD hole.

I have better visibility on the PC as to the “problem” delta T’s.  So pulses that are slightly too far apart, just on the boundary of what I consider to be valid, if I adjust my boundary accordingly, and now consider them valid, everything is peachy-keen.  I want to add a real-time option in my java client to allow this to be adjusted on the fly.  See problems? Adjust the slider, and problems go away.  Pretty neat.

I didn’t have this visibility when the microcontroller was interpreting the delta T’s.  The microcontroller had no easy feedback method to tell me what was happening.  Having high-level debugging of the information on the PC makes this all possible.

Nice to see the software purring.  There is still plenty of improvements to make, usability to be improved, etc.  But its working very nicely.

characterizing speed performance of floppy drive controller

So I’ve got things working rather swimmingly right now.  Switched drives from Samsung to Sony, and it’s made a huge difference.  The Sony just seems to work better.

I’m averaging about 355ms per track, yielding 57s total disk times.  The 355ms is made up of 313ms of transfer time at an effective throughput rate on the serial of around 1.175mbps.  Which is basically 1.5mbps baud rate, theoretical max of 1.2mbps.  This isn’t horrible performance, but I really want to get back to 2mbps.  I haven’t been using 2mbps because I have massive errors, but I think that there is some round off happening in my UART that prevents it from working correctly.  I need to revisit my UART code and find out exactly why 2mbps doesn’t work.  I’ve run this usb->ttl converter at 2mbps with my uC, so it really should work fine.

If I go to 2mbps, I’ll EASILY chop off the 88ms from 313ms, and I’ll be transferring the track to the PC in REAL TIME.  Basically, as fast as I receive it, I’ll be sending it to the PC.  Remember, that because I transmit the pulse times, and not the data, that fast times are really required.  This is a little more complicated than just saying the RAW MFM rate is 500kbps, so you need 500kbps of bandwidth to the PC.

There are several optimizations I can do, and I’ll post more later.

working FPGA version of the amiga floppy project

So, I’ve been working on the FPGA version of the amiga floppy project for some time.  I just recently had a fair bit of free time, and so everything came together rather quickly!

I’m now able to read amiga floppy disks in using the same Java client software I had developed for use with the Parallax SX microcontroller board.  There were a few minor changes in the software — most notably the read data routine from the hardware.

I’ve written the code in Verilog on a Xilinx Spartan-3e evaluation board.

The various hardware parts I’ve described:

  • UART: Written from scratch, a transmitter and a receiver.   Simple to use, variable baud rates.
  • FIFO: Generated from Xilinx’s CoreGen. This connects the floppy interface to the USB interface. 32k bytes
  • FSM to always empty the FIFO to the PC.  Once something goes in the FIFO, it immediately gets sent to the PC
  • Read-floppy-FSM: Stores 225ms of Delta T’s (aka time between edges) as 8-bit integers into the FIFO.
  • Command FSM: Receives single-character commands from the java software to execute (R for read, U for upper head, L for lower head, etc)
  • Transmit test pattern routine: Sends 32k of data to the PC to test for reliable communication

A couple advantages with the FPGA solution:

  • We transmit the data to the PC as soon as it’s available.  I want to characterize the actual latency, but it should be pretty small.  This is different from my load->FRAM, and then FRAM-> PC method.  This method should be much faster and we’re just not idling for 225ms.
  • Instead of transmitting the bit-sync’d raw MFM to the PC, I’m sending the delta T’s.  While this requires a little more processing on PC, the PC can more easily determine why a particularly sector can’t be read.  For instance, is the time between pulses too small? Too large?  On a fringe edge?  Plus, since the Java decodes these, I can now add sliders for “acceptable delta T’s” for each 4, 6, or 8us windows.  Before that would require modifying the firmware on the microcontroller.  I can also start to do statistical analysis on the pulse times.

I am currently doing about 430ms per track.  This sucks.  I was beating that by 100ms with my microcontroller.  However, the problem is that because a variable amount of data is going to the PC, the PC receiving code does not know when exactly to stop receiving, so there’s a wait-timer which I have to optimize.  Once I receive the minimum amount of data, I wait 100ms since the last received data, and then exit.  I’ve got to put my logic analyzers in place and figure out how to optimize it.

Denis@h3q can read a disk in 43s, which is pretty darn good.  He is using tokens like I mentioned here and here and here.  I’m transferring much more data though, which gives me more information.  I like his time, and maybe that would be a nice goal to beat!  Wow. That’s only 269ms/track.  Hrrrmm…. that’s pretty good.

Histogram of one track

Here’s a histogram of one track off an amiga floppy disk.  Notice the peaks near 4us, 6us, and 8us.

histo788

I’ve got this data from my logic analyzer, crunched through my little C proggie, and then graphed with Excel.

This shows roughly 45,000 delta Ts, the time between each transition.  The first group, of course, the ’10’s, second ‘100’s and the third, ‘1000’s.

got build environment working again, fixed uart code

So awhile ago I bought a quad core machine which runs Vista 64.  Once I had the new machine, I tried getting my build environment for the AFP working again.  NetBeans, the java IDE I use, has 64-bit support but there were a host of issues regarding FTDI drivers, jd2xx, etc which I fought over and eventually gave up.  I was getting error messages like “Can’t load IA 32-bit .dll on a AMD 64-bit platform” and there was a serious conflict between 32 bit and 64 bit JVM, JDK, DLL’s etc etc.  Pain in the butt.

I’ve had some time to work on stuff the last couple days and sit down and re-attack the problem.  I did manage to solve it by uninstalling anything Java that is 64-bit. 🙂  I believe it was just the JDK and JVM.  I also had to reinstall NetBeans because it links itself to a JDK —- once it was uninstalled, NetBeans would literally not boot with a link.  I looked all over NetBeans for something that defines a “target.”  You know, something where I can say “I want to build applications for 64-bit” or “32-bit” or whatever.  I couldn’t find it.  I uninstalled NetBeans, reinstalled it (this time it detected and recognized the 32-bit JDK), and voila, my java client now loads, builds, and runs correctly!@#

I hooked up my AFP again, and attempted to image and disk, and there were major problems.  Do you remember this post? This time it actually wasn’t that bad.  Another time somehow one of my SX28 pins were fried.

I’ve always wanted to do an extended UART data transfer test.  I’ve never really done this and I think it has been a big source of problems from the beginning.  Even though I checked the UART carefully for cycle counts(and done this 239408 times), put it on the logic analyzer, and even had someone else review it, there must have been a mistake.  I was corrupting about 3-5 bytes for every 100,000.  Not tons, but enough to show up during a disk transfer.

I started out really looking into my UART.  When bytes were corrupted, they were corrupted in exactly the same way:

The first data bit that was a 1-bit was ignored, and the data only started being received after the next one bit.  Let me give an example:

Let’s say the correct byte was : dec 138, or binary 1000 1010.  It would be received as dec 10 or 0000 1010.

correct byte might be : dec 39 or binary 0010 0111. It would be received as dec 7, or 0000 0111.

correct byte might be: dec 166 or binary 1010 0110. It might be rx’d as dec 38, or 0010 0110.

Remember, this only happened as an exception.

I eventually tweaked my UART by shortening the delay between the start bit and the first data bit, and also the time between bits by 20ns.  I’m honestly not sure why that worked, and it was mostly found by trial and error. But it had a profound and instant effect.  I was running trials and counting the number of bad bytes per 655k of transfer.  I was anywhere between 33-42 bad bytes per 655k.  When I made the change, it jumped to 0 !!

As a matter of fact, I just finished sending 284 megabytes (or 2.84 gigabits) of traffic through it without a single bit error!  I think that’s pretty decent.  The funny thing, I fired up “calc” to do some quick math, and I think the cpu interruption, or disk access, or something, caused me to lose some data.  In the actual real client, it would have automatically retransmitted the track, so it’s not the end of the world.

Once I fixed the uart, it started reading disks correctly again.

I’m pretty happy to see this thing working again.  Maybe I’ll read in some more amiga floppies tonight!