format_n_write for faster disk-from-image creation?
It seems possible to shorten the time to make e.g. Linux boot floppies by combining the formatting and dd'ing from a disk image file, using the 82078's Format with Write command. Has any thought been given to implementing this in superformat? I think it would be a neat addition, and would be willing to do this work if I got some pointers describing where this would need to hook into the code, feasibility, etc. Thanks in advance. _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
On Sun, Jul 20, 2003 at 09:27:31PM -0400, Charles R. Anderson wrote:
It seems possible to shorten the time to make e.g. Linux boot floppies by combining the formatting and dd'ing from a disk image file, using the 82078's Format with Write command. Has any thought been given to implementing this in superformat? I think it would be a neat addition, and would be willing to do this work if I got some pointers describing where this would need to hook into the code, feasibility, etc.
IMO you would have to implement this in another utility than superformat, which is meant to be portable, and uses "standard" Linux ioctls to format and verify tracks. For example, the "format and write" capability may not be available on all controllers (like cheap USB chips). Anyway to do that you would have to use FDRAWCMD (or implement another ioctl in the floppy driver), and while you are at it there are other things to do : - verifying while being on the same track (faster) and using raw cmds (avoids read-ahead, track changes, retries on errors, *way* faster) - write patterns & verify before writing+verifying actual data, also using raw cmds, and optionnaly immediately discarding bad disks. -- Julien Plissonneau Duquène. _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
On Mon, Jul 21, 2003 at 12:10:05PM +0000, Julien Plissonneau Duquene wrote: jpd> IMO you would have to implement this in another utility than jpd> superformat, which is meant to be portable, and uses "standard" Linux jpd> ioctls to format and verify tracks. For example, the "format and jpd> write" capability may not be available on all controllers (like cheap jpd> USB chips). Well, I would see that as being a waste to reimplement the entirety of superformat in another utility just to add one feature. If the format and write capability isn't there, it could either error out or fall back to a normal format without data. Or a symlink could be used so that if superformat is called as that symlink, it behaves in the new way. jpd> Anyway to do that you would have to use FDRAWCMD (or implement another jpd> ioctl in the floppy driver), and while you are at it there are other jpd> things to do : format_track currently uses the FD_FORMAT FDRAWCMD. I'd probably start by copying this function to a new one format_track_with_data and modify that to use the FDRAWCMD FD_FORMAT_N_WRITE, which takes a sector of data as an additional parameter. No additional kernel support should be needed, so the utility should remain portable across Linux kernels. jpd> - verifying while being on the same track (faster) and using raw cmds jpd> (avoids read-ahead, track changes, retries on errors, *way* faster) jpd> - write patterns & verify before writing+verifying actual data, also jpd> using raw cmds, and optionnaly immediately discarding bad disks. I was thinking of using MD5 to compare the input file to the data written. This could be done for each track-sized chunk of data, either during or after formatting, just like the --verify-later option does. This would bring yet another advantage to a utility like this, since currently one has to format/verify, then dd, then verify the data with dd/diff/md5sum, all separately. I don't know about you, but I am sick of making Linux boot disks that fail to boot because they failed to write properly, even after passing the format verify. _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
Well, I would see that as being a waste to reimplement the entirety of superformat in another utility just to add one feature. If the format and write capability isn't there, it could either error out or fall back to a normal format without data. Or a symlink could be used so that if superformat is called as that symlink, it behaves in the new way.
I agree, as long as the behavior is fully backward-compatible.
jpd> Anyway to do that you would have to use FDRAWCMD (or implement another jpd> ioctl in the floppy driver), and while you are at it there are other jpd> things to do :
format_track currently uses the FD_FORMAT FDRAWCMD. I'd probably start by copying this function to a new one format_track_with_data and modify that to use the FDRAWCMD FD_FORMAT_N_WRITE, which takes a sector of data as an additional parameter. No additional kernel support should be needed, so the utility should remain portable across Linux kernels.
So FDRAWCMD FD_FORMAT_N_WRITE already exists? If so that helps tremendously, as no kernel mods are needed. Thank Alain for that one.
I was thinking of using MD5 to compare the input file to the data written. This could be done for each track-sized chunk of data, either during or after formatting, just like the --verify-later option does. This would bring yet another advantage to a utility like this, since currently one has to format/verify, then dd, then verify the data with dd/diff/md5sum, all separately.
Why not just compare the data directly? Especially if you are doing a format-write-read pass?
I don't know about you, but I am sick of making Linux boot disks that fail to boot because they failed to write properly, even after passing the format verify.
Not a bad idea, though please keep in mind that it won't be quite as reliable as doing a full separate read pass with separate seeks. Superformat already compromised this area by combining formats and reads without seeking in between, because it saves a fair amount of time, but you need to keep in mind this increases the chance of uncaught errors somewhat. --------------------------------------------------------- David C. Niemi (Reston, Virginia, USA) niemi@tuxers.net Know the difference between the color of the wine and the color of the glass. (Jalaluddin Rumi) --------------------------------------------------------- _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
On Mon, Jul 21, 2003 at 09:51:02AM -0400, Charles R. Anderson wrote:
format_track currently uses the FD_FORMAT FDRAWCMD. I'd probably start by copying this function to a new one format_track_with_data and modify that to use the FDRAWCMD FD_FORMAT_N_WRITE, which takes a sector of data as an additional parameter. No additional kernel support should be needed, so the utility should remain portable across Linux kernels.
OK, I thought it used FDFMTTRK.
jpd> - verifying while being on the same track (faster) and using raw cmds jpd> (avoids read-ahead, track changes, retries on errors, *way* faster) jpd> - write patterns & verify before writing+verifying actual data, also jpd> using raw cmds, and optionnaly immediately discarding bad disks.
I was thinking of using MD5 to compare the input file to the data written. This could be done for each track-sized chunk of data, either during or after formatting, just like the --verify-later option does. This would bring yet another advantage to a utility like this, since currently one has to format/verify, then dd, then verify the data with dd/diff/md5sum, all separately.
I don't know about you, but I am sick of making Linux boot disks that fail to boot because they failed to write properly, even after passing the format verify.
Bad disks make through the format verify because the verify step is done using standard read and writes (it seems), and the kernel will do several retries when there is an error - without having a mean to tell the format/verify utility that there were retries. The other disadvantage of this is that the read-ahead feature mixes very badly with bad sectors : sectors being verified / bad sectors / / ...----v----XX----... =========... (length of read ahead) In this example, if one dash means a track and the verify is track by track, the kernel will still attempt to read several tracks ahead, and block on bad sectors at every track read. They will not fail (as long as you do not ask for the bad tracks), but they will be dog slow and painfully noisy. The same problem arises for CDs and HDDs. So I would also implement the verify step with raw commands, because it seems to be the only reliable way to avoid retries and read-ahead : more reliable AND faster. In my experience there is no difference between formatting, writing & verifying patterns, writing & verifying data in one pass (not changing tracks between steps) or with recalibration between steps -- excepted when it comes to bad drives/controllers. I hope this helps ... -- Julien Plissonneau Duquène. _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
It might be worth considering, though a fair number of PCs may be missing this command (at least enough that it would have to be checked for). Then again, the kernel does already test to see whether the FDC is an 82078 or something less. Another reason to consider *not* doing it would be that on dodgy floppies, formatting and writing as separate steps might have a better chance of detecting errors related to seeking; which also applies to writing and reading back the same data. But that is probably a somewhat infrequent and dubious benefit. So perhaps format+write+read (verify) of both sides before seeking to the next track would make sense, if time is of the essence, and especially if validity of the floppies is not the top concern. The cleanest way to add this would be to add a switch to superformat, which would be automatically ignored (with a warning) on pre-82078 FDCs. I believe the method used by superformat to do what it does now would also work for what you want to do -- it shouldn't take a dramatic rewrite. If it turns out to work well, I think it would be worth adding into the standard superformat, though Alain has the final word on that. DCN On Sun, 20 Jul 2003, Charles R. Anderson wrote:
It seems possible to shorten the time to make e.g. Linux boot floppies by combining the formatting and dd'ing from a disk image file, using the 82078's Format with Write command. Has any thought been given to implementing this in superformat? I think it would be a neat addition, and would be willing to do this work if I got some pointers describing where this would need to hook into the code, feasibility, etc.
Thanks in advance.
_______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
-- --------------------------------------------------------- David C. Niemi (Reston, Virginia, USA) niemi@tuxers.net Know the difference between the color of the wine and the color of the glass. (Jalaluddin Rumi) --------------------------------------------------------- _______________________________________________ fdutils mailing list fdutils@tux.org http://www.tux.org/mailman/listinfo/fdutils
participants (3)
-
Charles R. Anderson -
David C Niemi -
Julien Plissonneau Duquene