Jun 22

more SX to PC tests

Although I think I’ve long had this working good, I did some more tests just to verify.

Latest numbers:

Each cycle is about 15.8us, 2us per bit, 8 bits, this sounds fine. This does confirm, however, what I constantly see about the data rate being actually a little bit higher than 500kbps for the raw amiga MFM. Some other numbers from today 3.9 (for 4us), 5.88(for a 6us), and 7.64(for a 8us). That 8us one is always low! I’ve seen it hover around 7.5 before!

The PC is reacting quickly, and total worst case I saw today was about 9us for the whole cycle.

Jun 21

floppy.c’s mfm decoder

A couple days ago, whenever I got ahold of floppy.c, I wrote a quickie code stub that utilizes floppySectorMfmDecode from floppy.c of fellow (http://fellow.sourceforge.net). Note you must use the mirrors to actually download anything….

This MFM decoder from fellow.c is pretty clear.  AFR is located here See http://www.techtravels.org/amiga/afr10f.zip for the local copy.

Anyways, I used mainly the data-portion of the decoder, which was putting out junk for my input from the SX/amiga drive. So I sort of put it on the back burner.

Then I realized that since it decodes the header, this might be a GREAT check for SOME data validity. Some of the header fields are MFM-encoded separately from each other, and from the actual data. What this means is that you need less good data to get a valid read. What sucks is that the actual data is spread out across 1024 bytes of MFM encoding, if the second half is screwed up you lose the even bits of your data. If the first half you lose the odd bits. This is because the clock bits take up the other ‘half’ of encoded data….

Here’s the results of that code stub:

(2372) Format: 255 Sector: 9 Track: 0 Sectors to end: 9
(5620) Format: 255 Sector: 1 Track: 0 Sectors to end: 6
(6700) Format: 170 Sector: 245 Track: 85 Sectors to end: 232 (not valid format)
(8811) Format: 170 Sector: 65 Track: 85 Sectors to end: 240 (not valid format)
(20645) Format: 84 Sector: 170 Track: 174 Sectors to end: 193 (not valid format)
(28232) Format: 255 Sector: 3 Track: 2 Sectors to end: 10
(35714) Format: 255 Sector: 10 Track: 2 Sectors to end: 82
(70992) Format: 170 Sector: 93 Track: 64 Sectors to end: 245 (not valid format)
(89788) Format: 255 Sector: 7 Track: 6 Sectors to end: 4
(98245) Format: 255 Sector: 1 Track: 7 Sectors to end: 3

The valid sectors have a format ID of 255, 0xFF, the other ones are simply corrupted data.

You can even tell they are invalid because the sector numbers, track numbers etc are out of range.

So what’s this mean? I gotta get to fine tuning my SX code that captures this data. I knew I wasn’t SUPER close because out of 100kb of code, I’m getting a lousy 10 sectors, where I should be getting much more.

For this, I really have to learn exactly how the RTCC functions in fine detail, because doing this correctly requires very good timing.

Jun 20

test results of my output file

What I wanted to know was if the output file that I collected via my setup was actually correct. I was confident that since I was getting 4489′s consistently, that I have to be at least part way there…….

The format of my output file should only contain the proper groupings of acceptable amiga MFM encodings, and those possibilities are:

10, 100, or 1000. So 2-bit, 3-bit, and 4-bit combinations.

Nothing else is acceptable and nothing else has been seen via the scope hooked up to the output.

I already had my file created, but it contained binary(read: vs text) 8-bit characters and I usually view this in HEX, but since I’m looking for binary(read: 0 vs 1) results that are all strung together, including across byte boundaries, then I have to look at a lower level. Here’s what I did:

I created a small program that reads in my original file byte for byte, and then output to a different file the binary equivalent, so a ’0xAA’ input gives 10101010 output.

Then, I needed some way to be able to detect the groupings, but trying to do this programatically is a real pain in the butt. I couldn’t come up with a good way to do this. Perhaps some perl experts who really know how to manipulate some regex might come up with a way…..

What I did was to load the output which is now one large ASCII text file containing 1′s and 0′s (ASCII 48′s and 49′s) into Microsoft Word. Then I did a search and replace for the various cases, replacing them with NULL.

Search for “1000″, yielded 3,047 cases, or 12188 bits worth, or 1523.5 bytes worth.
Search for “100″, yielded 15,136 cases, or 45408 bits worth, or 5676 bytes worth.
Search for “10″, yielded 34,594 cases, or 69188 bits worth, or 8648.5 bytes worth.

Add these all up, and you get 52,777 groupings yielding 15848 valid BYTES of raw MFM encoding.

The total filesize was 15892 bytes, so that means that there are 44 mal-formed MFM bytes, or 352 bad bits. These would be cases where there were too many zeros trailing a one, or a one immediately following another one. These are invalid cases, that really shouldn’t exist.

I guess there are a couple possibilities

1. Maybe there are other cases(that I have yet to recognize) that come up much less often, namely 1+(4 zeros), 1+(5 zeros), 1+(6 zeros), and 1+(7 zeros), also 11111111. If I do the same search and replace and add these cases, my file ends up NULL. Perfectly. No bits off. I can’t believe my total solution is perfect — I can’t believe I’m not missing at least a bit here or there.

2. Maybe my hardware is slipping. I’ve long thought that even the new setup, although yielding SOME accurate results has got to be getting of out sync, if only for a group or two……

Jun 17

an output file from my project

OK. Here’s the dealio:

I was using the MFMeditor portion of DiskMonTools on the amiga to read a raw sector. Now the darn editor won’t save to disk, so I can’t COMPARE the raw data which I’d love to do. I’m half contemplating taking some digital pictures and OCR’ing the fuqrs just so I can get some data to compare.

This is exactly one track’s, track 0, according to DiskMonTools. Don’t ask me why its not 13800, or 11xxx, etc.

The point right now isn’t so much that I’m getting one track perfectly delimited, its that what comes out of the drive is being read properly by my solution.

DiskMonTools says it’s reading 16200 bytes. My project is reading a filesize of 15892 consistently. I don’t think the data output is exactly the same each time, and that’s a problem Im going to have to address eventually.

I would rather be reading 16200, but hell i’d rather be under than over. This means my SX routines are missing some data someplace, if I had to bet, its an Interrupt problem, where a regular RTCC rollover event is happening during an edge event.

Here’s the link to the sampled file.

I think the sector header isn’t at the beginning of the file because I think the amiga starts streaming data immediately out of the drive (well, after it spins up) without regard to the position within the track.

Jun 16

will put up sample files

I’m going to put up a sample file or two, but I wanted to get the smallest complete group of bytes that I can…. ie like one full sector, etc.

So far, I’m just basically streaming bytes, via disk copy software, from the drive into memory on the amiga. I’m “sniffing” this connection via the daisy chain port, and have that connected to my SX controller. I start up my software right before the copy, and so I’m getting the first 100kb that comes out of the drive.

What I don’t know yet is this: Am I getting the correct amount of data that the drive is actually putting out? I’m going to use DMS or something that allows me to copy exactly one track, and then compare how much data I’ve received to what it should be….. This will be the true proof that I’m close.

I also want to fully decode, by hand (if possible), the MFM data — at least the full sector header, anyways. But this damn goofy amiga MFM spec is crazy, and so I might end up using a couple small routines to do it. I don’t fully understand how it works, but I *do* have the necessary documentation, etc

I’ll have stuff up by Saturday…..

Apr 10

too many variables

I think that’s really the problem I’ve been facing with this floppy project. Trials and tribulations indeed. Or at least trial and error, with stress on the error part. There are probably less than half-a-dozen people worldwide that have the very specific knowledge that I need to get this off the ground. These would be people from CAPS, one person who sells an amiga (as well as other floppy drives) floppy controller card, and that’s it.

I’ve talked to some really nice ex-Commodore employees who have been friendly, and helpful.

The problem, of course, is that these machines were created a long time ago. The programmers of the original OS, freeware/shareware software, hardware, have all long forgotten the details. And you know what they say, “the devil’s in the details.”

To give you an idea of the variables with which I’m working:

1. Am I reading the hardware signal right? Are there *really* only three possibilities coming out of the drive, a “10″, “100″, and a “1000″? This is all I’ve been able to observe…

2. Is the software I’ve written on the SX microcontroller properly sampling this data at the right time? I have two interrupts occuring, one for the edge detection and one for the 2us timeout which clocks a zero in (up to 3 of them in a row before going idle). So far, sometimes this works, and sometimes it doesn’t. Why and how is this getting out of sync?

3. Is the SX transmitting faster than the PC can handle this? So far, my observations say no, and I implemented a software FIFO to help out.

4. Is my software FIFO working?

5. Is my PC software, that’s designed to receive all this, working properly? I’m now storing the transmitted bytes in a memory array, and later storing to disk, to prevent any speed issues associated with accessing the hard drive.

The REAL problem here is that I simply don’t know what is actually leaving the drive in terms of data. The only thing I’ve been able to figure out is that the MORE data is leaving the drive than is showing up on the PC. This is a bad sign. Where’s the data being lost? My guess, the sampling isn’t working properly. Something is slipping. But how the heck can it slip when a transition resets it, and transition is occuring on a very regular basis (minimum every 4us, maximum every 8us). The thing only has to run freely for a few cycles.

Add the fact that the only real pattern that I can VISUALLY look for is the 0xAAAA 0xAAAA 0×4489 0×4489 raw sync sector in raw MFM. Any words, or real data has to be decoded before, properly aligned on a byte, etc. PAIN IN THE BUTT to figure out if everything is actually working. Most of my programming projects, whatever they might be, are straight-forward. Run it, see if it works, and then go back and fix it if it doesn’t. Even the indication that this thing is working is obscured.

There are so many variables, and I’m constantly changing all of them, that who knows when one part is working? Of course, you say, only change one thing at a time. OK, makes sense, but this just isn’t practical. If I change how the SX xfers to the PC, then I have to modify the PC software accordingly…. So if I make a mistake in coding on one side or the other, who knows?

Maybe I’m lacking the necessary basic project management, coding, microcontroller, hardware experience required to get this off and running?

BTW: my title of this entry reminds me of “Too many secrets” or “Cootys Rat Semen.” from Sneakers. If you don’t know what I’m talking about, please disregard! :)

Mar 29

updates

I originally thought that the SX might be sending the data too fast for the PC to handle. As my last post mentioned, one full cycle is 12us, but who knows if this is during a disk-write, going into the cache, etc. I took several samples and they were all around 12us but I wanted to be sure that the PC wasn’t dropping data on the floor. The SX, up ’til now, didn’t have support for handling this situation.

I made a couple improvements on the SX side and the PC side. On the SX, I added a 16-byte software FIFO, so if the PC hasn’t ack’d the last byte yet when a new byte is ready, it simply gets thrown into an array, and sent to the PC as soon as it’s ready. It takes 16us(2us * 8 bits) minimum for the SX to get 1 one byte, and so a 16-byte FIFO gives me 256us of delay before this would be a serious issue. This is still 1/4000th of a second, so the PC can’t really dilly-dally, but it gives me some breather space.

On the PC side, instead of getting the byte from the parallel port and writing it directly to disk, I’m shoving it into an array first, and then writing at the end of the transfer. All of this software is of the barely-working variety….

I have yet to see anything intelligible to come from the drive which means my whole solution doesn’t function, yet. There are so many variables which makes this a tough nut to crack. Just the mere fact that the data coming from the drive is MFM encoded, simply looking at the data doesn’t tell you whether or not the raw data is correct. And the goofy MFM encoding that the Amiga uses makes it even tougher. BUT — I’m still holding out that I should be able to see the SYNC words. Of course, the problem there is that the data can be shifted, and not on even byte boundaries, so you have to shift the data up to 8-bit locations in order for that to be readable.

My next step is to run DiskMonTools, an amiga program, which will display the raw MFM from a floppy on screen. I’m hoping I can sort of learn what some common encoded patterns are, and look for them in the output from my program. Things like headers etc have lots of static data, so they are easier to identify.

I’m hoping to see some progress one of these days……

Mar 18

Timing problems still?

I think I’m still having timing related issues. I have RTCC overflowing every 1.8us which I *know* is less than the 2us bit cell(or less than half the bit cell, if you take it to be 4us), but this seems to be what I need to do in order to hit the center of the next bitcells. I think this is probably because it takes a little while to detect and process the edge transition, and by that time, I’m too far into the 1st cell to use a full 2us jump. I contemplated (and did in previous designs) use a smaller delay after the transition than after a non-transition. This does give the desired effect, so perhaps I’ll put that into play.

You know, sync isn’t actually too big of an issue, because the edge transitions are so frequent, that I’m never going more than 3 bitcells(ie 3 two-microsecond cells) in a row in a free-running mode. And since the data always starts with a transition, I don’t have to worry about getting out-of-sync. If I happen to lose a bit or two, besides the shifting-problem, my device will just wait and resync at the next bitcell.

I think I’m accurately capturing, finally, on the PC the exact stream coming out of the drive. This is good, but now it sort of brings me to the next phase of work on the PC trying to get the PC to massage the raw MFM data into actual data. This isn’t easy (for me) although I pretty much have all the actual documentation I need.

I’m still concerned I’m doing something wrong because the output I’m getting from the drive is very heavy in 0×55 and 0xAA’s. The AA’s might make sense because they are the first-half of the sync words(0xAAAA 0xAAAA 0×4489 0×4489) so that might make sense, but the 0×4489′s are nowhere to be found. 75% of some samples are 0×55 and 0xAA’s. This doesn’t sound right. I would expect a much wider character distribution.

I’ve been trying to use Marco’s code to do the MFM decoding, and although it compiled easily, I just don’t think it’s going to work ‘off the shelf’ like that.

Mar 13

making progress on the MFM hardware decoding

The whole question I’ve had for the last couple days (and potentially the last couple months) is how do you turn the signal coming out of the floppy drive into 0′s and 1′s. This really isn’t MFM ‘decoding’ perse, that happens once we have the raw MFM data, and that process is actually more convoluted than the current issue.

With the help of Michael C from the parallax forums, it’s all starting to come together. Although the details for me are a little fuzzy, there are three possible states coming out of the drive. Take a look at the attached picture, which explains the three possible states coming out of the drive.

I’ll explain more as I understand more, but this is the generic explanation…….

MFM amiga decoding

Mar 11

Am I reading MFM wrong?

I’m beginning to think that I don’t fully understand the signalling coming out of the drive. I understand the concepts behind MFM (I think), but the actual application to this project, I’m not so sure.

As far as I can tell, I’ve got the code done properly on the SX to do what I want it to do. I take this information into the PC app I wrote, and I’m not getting anything intelligible. At the minimum, I should be seeing the Sync pattern (0xAAAA 0xAAAA 0×4489 0×4489) from the drive. The amiga puts this pattern at the beginning of a sector header. And this is raw MFM — so even if my decoding is wrong, I should be seeing this pattern, but I’m not.

I’m reading it wrong in hardware. What I think constitutes a 1 or 0 in terms of a signal, obviously does not. I find plenty of generic references online to MFM, and I have read them — but very few that actually show me a scope image. They have their lame handdrawn pictures but none show a real application.

I’ve bumped my scope from 2us to 5us just so I can look at a little more of the signal. I’m starting to see a pattern where ‘transition to 0, short positive’ or ‘transition to zero, long positive’ might be my two different states, but I dunno.

Time to read about MFM again……

More scope images