Linux 4K drive formatting
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server. Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there. Should I worry that the drive reports fdisk -l /dev/sdc Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ...instead of 3 TB, and thinks the sector and I/O size is 512 bytes? Advice, please! Kevin
For starters, are we talking about the possibility of 4 KB sector size instead of 512 bytes? I don't have Linux-specific (or even Unix-specific) knowledge on this, but I sure know about this topic overall and have strong thoughts. That asinine 512 byte sector size that we got from the original IBM PC-XT around 1982-1983 has been a huge space waster and efficiency killer ever since! It truly ought to be relegated to the trash heap (of history). In fact, the concept of a cluster in the DOS/Windows FAT file systems was only created for this reason, and it only partially offset the problem (it doesn't address the space wastage). Even with a perfect OS, many of the older hard disk controllers were themselves retricted to 512 byte sectors. Even the older versions of DOS and Windows had settable parameters for media sector size, but forums pointed out that these had never been properly tested. Interestingly, the Fujitsu magneto-optical drives that I handled in the 1990's and early 2000's used multi-KB sectors (as did all other magneto-optical drives), and came with drivers and/or software to "make them work" with DOS/Windows. So, I encourage larger sectors, and hope that the current Linux world handles them well. I should think that modern SATA and SCSI hardware controllers (especially the SATA controllers on the disks) can handle this. Hartmut Sager On 19 January 2013 17:54, Kevin McGregor <kevin.a.mcgregor@gmail.com> wrote:
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server.
Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there.
Should I worry that the drive reports fdisk -l /dev/sdc
Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
...instead of 3 TB, and thinks the sector and I/O size is 512 bytes?
Advice, please!
Kevin
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Oh, one more thing. The cylinders/heads/sectors legacy stuff is irrelevant today. Even the old (4-entry) partition table also had a pair of fields for starting position (displacement from sector 0) and size of partition, and on later PC's, these two parameters alone were used during boot and then for the booted OS to know where the partitions are. By the time we got LBA (large/logical block addressing), this practise was already the case. Hartmut Sager On 19 January 2013 17:54, Kevin McGregor <kevin.a.mcgregor@gmail.com> wrote:
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server.
Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there.
Should I worry that the drive reports fdisk -l /dev/sdc
Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
...instead of 3 TB, and thinks the sector and I/O size is 512 bytes?
Advice, please!
Kevin
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Short answer: yes, you should be worried. However, I would double-check that output against both dmesg and the contents of /proc/partitions just in case there’s a 2TB limit built into fdisk. (Actually, now that I think of it, there is. You have to use [g]parted to make a 3TB partition, IIRC.) -Adam Thompson athompso@athompso.net (204) 291-7950 - direct (204) 489-6515 - fax From: roundtable-bounces@muug.mb.ca [mailto:roundtable-bounces@muug.mb.ca] On Behalf Of Kevin McGregor Sent: Saturday, January 19, 2013 5:54 PM To: Continuation of Round Table discussion Subject: [RndTbl] Linux 4K drive formatting So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server. Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there. Should I worry that the drive reports fdisk -l /dev/sdc Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ...instead of 3 TB, and thinks the sector and I/O size is 512 bytes? Advice, please! Kevin
Oh… and if it’s not just an fdisk problem, you can contact WD support directly for RED drives only: <http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred> http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred -Adam Thompson athompso@athompso.net (204) 291-7950 - direct (204) 489-6515 - fax From: roundtable-bounces@muug.mb.ca [mailto:roundtable-bounces@muug.mb.ca] On Behalf Of Kevin McGregor Sent: Saturday, January 19, 2013 5:54 PM To: Continuation of Round Table discussion Subject: [RndTbl] Linux 4K drive formatting So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server. Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there. Should I worry that the drive reports fdisk -l /dev/sdc Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ...instead of 3 TB, and thinks the sector and I/O size is 512 bytes? Advice, please! Kevin
In dmesg, it shows [ 11.317498] scsi 6:0:1:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.319897] sd 6:0:1:0: Attached scsi generic sg3 type 0 [ 11.323370] sd 6:0:1:0: [sdc] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB) [ 11.325746] scsi 6:0:2:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.328162] sd 6:0:2:0: Attached scsi generic sg4 type 0 [ 11.330410] sd 6:0:2:0: [sdd] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB) In /proc/partitions: 8 32 2147483647 sdc 8 48 2147483647 sdd Using parted: $ sudo parted /dev/sdc GNU Parted 2.2 Using /dev/sdc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Error: /dev/sdc: unrecognised disk label (parted) mklabel gpt (parted) print Model: ATA WDC WD30EFRX-68A (scsi) Disk /dev/sdc: 2199GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags It looks like the kernel is not recognizing the size correctly. So... Upgrade to Ubuntu Precise 12.04 LTS? On Sat, Jan 19, 2013 at 11:19 PM, Adam Thompson <athompso@athompso.net>wrote:
Oh… and if it’s not just an fdisk problem, you can contact WD support directly for RED drives only: http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred ****
** **
** **
-Adam Thompson****
athompso@athompso.net****
(204) 291-7950 - direct****
(204) 489-6515 - fax****
** **
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor *Sent:* Saturday, January 19, 2013 5:54 PM *To:* Continuation of Round Table discussion *Subject:* [RndTbl] Linux 4K drive formatting****
** **
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server.****
** **
Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there.****
** **
Should I worry that the drive reports****
fdisk -l /dev/sdc****
** **
Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes****
255 heads, 63 sectors/track, 267349 cylinders****
Units = cylinders of 16065 * 512 = 8225280 bytes****
Sector size (logical/physical): 512 bytes / 512 bytes****
I/O size (minimum/optimal): 512 bytes / 512 bytes****
** **
...instead of 3 TB, and thinks the sector and I/O size is 512 bytes?****
** **
Advice, please!****
** **
Kevin****
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Well, I upgraded to 12.04 and of course there's no change since the LSI SAS1068E controller doesn't support drives larger than 2 TB. Sigh. Can anyone recommend a 2- or 4-port PCIe non-RAID controller that works under Linux, supports large drives and is affordable? Kevin On Sun, Jan 20, 2013 at 8:02 AM, Kevin McGregor <kevin.a.mcgregor@gmail.com>wrote:
In dmesg, it shows [ 11.317498] scsi 6:0:1:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.319897] sd 6:0:1:0: Attached scsi generic sg3 type 0 [ 11.323370] sd 6:0:1:0: [sdc] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB) [ 11.325746] scsi 6:0:2:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.328162] sd 6:0:2:0: Attached scsi generic sg4 type 0 [ 11.330410] sd 6:0:2:0: [sdd] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB)
In /proc/partitions: 8 32 2147483647 sdc 8 48 2147483647 sdd
Using parted: $ sudo parted /dev/sdc GNU Parted 2.2 Using /dev/sdc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Error: /dev/sdc: unrecognised disk label (parted) mklabel gpt (parted) print Model: ATA WDC WD30EFRX-68A (scsi) Disk /dev/sdc: 2199GB Sector size (logical/physical): 512B/512B Partition Table: gpt
Number Start End Size File system Name Flags
It looks like the kernel is not recognizing the size correctly.
So... Upgrade to Ubuntu Precise 12.04 LTS?
On Sat, Jan 19, 2013 at 11:19 PM, Adam Thompson <athompso@athompso.net>wrote:
Oh… and if it’s not just an fdisk problem, you can contact WD support directly for RED drives only: http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred ****
** **
** **
-Adam Thompson****
athompso@athompso.net****
(204) 291-7950 - direct****
(204) 489-6515 - fax****
** **
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor *Sent:* Saturday, January 19, 2013 5:54 PM *To:* Continuation of Round Table discussion *Subject:* [RndTbl] Linux 4K drive formatting****
** **
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server.****
** **
Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there.****
** **
Should I worry that the drive reports****
fdisk -l /dev/sdc****
** **
Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes****
255 heads, 63 sectors/track, 267349 cylinders****
Units = cylinders of 16065 * 512 = 8225280 bytes****
Sector size (logical/physical): 512 bytes / 512 bytes****
I/O size (minimum/optimal): 512 bytes / 512 bytes****
** **
...instead of 3 TB, and thinks the sector and I/O size is 512 bytes?****
** **
Advice, please!****
** **
Kevin****
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
One other thing… the IBM M1015 ServeRAID card can often be had cheaply, and although it is a RAID card, you can apparently flash the generic IT-mode LSI firmware onto it, turning it into a regular HBA. See <http://www.servethehome.com/ibm-serveraid-m1015-75-dollars/> http://www.servethehome.com/ibm-serveraid-m1015-75-dollars/ for details. -Adam Thompson athompso@athompso.net (204) 291-7950 - direct (204) 489-6515 - fax From: roundtable-bounces@muug.mb.ca [mailto:roundtable-bounces@muug.mb.ca] On Behalf Of Kevin McGregor Sent: Sunday, January 20, 2013 1:20 PM To: Adam Thompson; Continuation of Round Table discussion Subject: Re: [RndTbl] Linux 4K drive formatting Well, I upgraded to 12.04 and of course there's no change since the LSI SAS1068E controller doesn't support drives larger than 2 TB. Sigh. Can anyone recommend a 2- or 4-port PCIe non-RAID controller that works under Linux, supports large drives and is affordable? Kevin On Sun, Jan 20, 2013 at 8:02 AM, Kevin McGregor <kevin.a.mcgregor@gmail.com> wrote: In dmesg, it shows [ 11.317498] scsi 6:0:1:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.319897] sd 6:0:1:0: Attached scsi generic sg3 type 0 [ 11.323370] sd 6:0:1:0: [sdc] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB) [ 11.325746] scsi 6:0:2:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5 [ 11.328162] sd 6:0:2:0: Attached scsi generic sg4 type 0 [ 11.330410] sd 6:0:2:0: [sdd] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB) In /proc/partitions: 8 32 2147483647 sdc 8 48 2147483647 sdd Using parted: $ sudo parted /dev/sdc GNU Parted 2.2 Using /dev/sdc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Error: /dev/sdc: unrecognised disk label (parted) mklabel gpt (parted) print Model: ATA WDC WD30EFRX-68A (scsi) Disk /dev/sdc: 2199GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags It looks like the kernel is not recognizing the size correctly. So... Upgrade to Ubuntu Precise 12.04 LTS? On Sat, Jan 19, 2013 at 11:19 PM, Adam Thompson <athompso@athompso.net> wrote: Oh… and if it’s not just an fdisk problem, you can contact WD support directly for RED drives only: <http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred> http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred -Adam Thompson athompso@athompso.net (204) 291-7950 <tel:%28204%29%20291-7950> - direct (204) 489-6515 <tel:%28204%29%20489-6515> - fax From: roundtable-bounces@muug.mb.ca [mailto:roundtable-bounces@muug.mb.ca] On Behalf Of Kevin McGregor Sent: Saturday, January 19, 2013 5:54 PM To: Continuation of Round Table discussion Subject: [RndTbl] Linux 4K drive formatting So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server. Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there. Should I worry that the drive reports fdisk -l /dev/sdc Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes 255 heads, 63 sectors/track, 267349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ...instead of 3 TB, and thinks the sector and I/O size is 512 bytes? Advice, please! Kevin _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Thanks for the tip. I'll keep an eye out for a deal on those. Right now the cheapest one I could find on eBay is $96, from South Korea. In the meantime, I'm using the SATA 2 ports on the motherboard which has 4 ports and I only have 3 SATA drives right now. Not 6 Gbps, but it'll do for now. Kevin On Sun, Jan 20, 2013 at 1:44 PM, Adam Thompson <athompso@athompso.net>wrote:
One other thing… the IBM M1015 ServeRAID card can often be had cheaply, and although it is a RAID card, you can apparently flash the generic IT-mode LSI firmware onto it, turning it into a regular HBA.****
See http://www.servethehome.com/ibm-serveraid-m1015-75-dollars/ for details.****
** **
-Adam Thompson****
athompso@athompso.net****
(204) 291-7950 - direct****
(204) 489-6515 - fax****
** **
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor
*Sent:* Sunday, January 20, 2013 1:20 PM *To:* Adam Thompson; Continuation of Round Table discussion *Subject:* Re: [RndTbl] Linux 4K drive formatting****
** **
Well, I upgraded to 12.04 and of course there's no change since the LSI SAS1068E controller doesn't support drives larger than 2 TB. Sigh.****
** **
Can anyone recommend a 2- or 4-port PCIe non-RAID controller that works under Linux, supports large drives and is affordable?****
** **
Kevin****
** **
On Sun, Jan 20, 2013 at 8:02 AM, Kevin McGregor < kevin.a.mcgregor@gmail.com> wrote:****
In dmesg, it shows****
[ 11.317498] scsi 6:0:1:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5****
[ 11.319897] sd 6:0:1:0: Attached scsi generic sg3 type 0****
[ 11.323370] sd 6:0:1:0: [sdc] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB)****
[ 11.325746] scsi 6:0:2:0: Direct-Access ATA WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 5****
[ 11.328162] sd 6:0:2:0: Attached scsi generic sg4 type 0****
[ 11.330410] sd 6:0:2:0: [sdd] 4294967294 512-byte logical blocks: (2.19 TB/1.99 TiB)****
** **
In /proc/partitions:****
8 32 2147483647 sdc****
8 48 2147483647 sdd****
** **
Using parted:****
$ sudo parted /dev/sdc****
GNU Parted 2.2****
Using /dev/sdc****
Welcome to GNU Parted! Type 'help' to view a list of commands.****
(parted) print****
Error: /dev/sdc: unrecognised disk label****
(parted) mklabel gpt****
(parted) print****
Model: ATA WDC WD30EFRX-68A (scsi)****
Disk /dev/sdc: 2199GB****
Sector size (logical/physical): 512B/512B****
Partition Table: gpt****
** **
Number Start End Size File system Name Flags****
** **
** **
It looks like the kernel is not recognizing the size correctly.****
** **
So... Upgrade to Ubuntu Precise 12.04 LTS?****
** **
On Sat, Jan 19, 2013 at 11:19 PM, Adam Thompson <athompso@athompso.net> wrote:****
Oh… and if it’s not just an fdisk problem, you can contact WD support directly for RED drives only: http://support.wdc.com/contact/contact.asp?lang=en&ct=wdred ****
****
****
-Adam Thompson****
athompso@athompso.net****
(204) 291-7950 - direct****
(204) 489-6515 - fax****
****
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor *Sent:* Saturday, January 19, 2013 5:54 PM *To:* Continuation of Round Table discussion *Subject:* [RndTbl] Linux 4K drive formatting****
****
So... I picked up my lovely new WD Red 3 TB drives today and installed them in my home server.****
****
Now what? :-) Specifically, I'm running Ubuntu Server 10.04 LTS (2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux), and I'm wondering if I should upgrade to Precise before I do anything with the drives, or use them as-is. I know I should align the partitions at (at least) 4 KB boundaries, and 10.04 by default uses 1 MB alignment, so I should be okay there.****
****
Should I worry that the drive reports****
fdisk -l /dev/sdc****
****
Disk /dev/sdc: 2199.0 GB, 2199023254528 bytes****
255 heads, 63 sectors/track, 267349 cylinders****
Units = cylinders of 16065 * 512 = 8225280 bytes****
Sector size (logical/physical): 512 bytes / 512 bytes****
I/O size (minimum/optimal): 512 bytes / 512 bytes****
****
...instead of 3 TB, and thinks the sector and I/O size is 512 bytes?****
****
Advice, please!****
****
Kevin****
** **
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable****
** **
** **
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (3)
-
Adam Thompson -
Hartmut W Sager -
Kevin McGregor