Why VMware Virtual disk so slow (da0: 3.300MB/s transfers) ?

[Dongsheng Song]

On Wed, Aug 21, 2013 at 2:52 PM, Dongsheng Song
<dongsheng.song at gmail.com> wrote:
> I don't think this is a battery issue, since Linux guest in the same
> ESXi host give me very good results:
>
> root at CRM-SYSLOG:~# dd if=/dev/zero bs=8M count=128 of=/tmp/1G.img
> 128+0 records in
> 128+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.10386 s, 510 MB/s
>
> root at CRM-SYSLOG:~# dd if=/dev/zero bs=8M count=128 of=/tmp/1G.img
> 128+0 records in
> 128+0 records out
> 1073741824 bytes (1.1 GB) copied, 7.08881 s, 151 MB/s
>
> root at CRM-SYSLOG:~# dd if=/dev/zero bs=8M count=128 of=/tmp/1G.img
> 128+0 records in
> 128+0 records out
> 1073741824 bytes (1.1 GB) copied, 6.85367 s, 157 MB/s

When I install FreeBSD 9.1 in the same ESXi guest, I got at least 20 MB/s:

root at CRM-FB9:/root # dd if=/dev/zero bs=8m count=128 of=/tmp/1G.img
128+0 records in
128+0 records out
1073741824 bytes transferred in 34.246246 secs (31353563 bytes/sec)

root at CRM-FB9:/root # dd if=/dev/zero bs=8m count=128 of=/tmp/1G.img
128+0 records in
128+0 records out
1073741824 bytes transferred in 17.962520 secs (59776793 bytes/sec)

root at CRM-FB9:/root # dd if=/dev/zero bs=8m count=128 of=/tmp/1G.img
128+0 records in
128+0 records out
1073741824 bytes transferred in 50.373031 secs (21315807 bytes/sec)


root at CRM-FB9:/root # dmesg
(cd0:ata1:0:0:0): READ(10). CDB: 28 0 0 0 7f aa 0 0 1 0
(cd0:ata1:0:0:0): CAM status: SCSI Status Error
(cd0:ata1:0:0:0): SCSI status: Check Condition
(cd0:ata1:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(cd0:ata1:0:0:0): Error 6, Unretryable error
(cd0:ata1:0:0:0): cddone: got error 0x6 back
g_vfs_done():iso9660/FREEBSD_INSTALL[READ(offset=66932736,
length=2048)]error = 6
(cd0:ata1:0:0:0): READ(10). CDB: 28 0 0 0 93 c0 0 0 1 0
(cd0:ata1:0:0:0): CAM status: SCSI Status Error
(cd0:ata1:0:0:0): SCSI status: Check Condition
(cd0:ata1:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(cd0:ata1:0:0:0): Error 6, Unretryable error
(cd0:ata1:0:0:0): cddone: got error 0x6 back
g_vfs_done():iso9660/FREEBSD_INSTALL[READ(offset=77463552,
length=2048)]error = 6
(cd0:ata1:0:0:0): READ(10). CDB: 28 0 0 0 93 c0 0 0 1 0
(cd0:ata1:0:0:0): CAM status: SCSI Status Error
(cd0:ata1:0:0:0): SCSI status: Check Condition
(cd0:ata1:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(cd0:ata1:0:0:0): Error 6, Unretryable error
(cd0:ata1:0:0:0): cddone: got error 0x6 back
g_vfs_done():iso9660/FREEBSD_INSTALL[READ(offset=77463552,
length=2048)]error = 6
(cd0:ata1:0:0:0): READ(10). CDB: 28 0 0 0 93 c0 0 0 1 0
(cd0:ata1:0:0:0): CAM status: SCSI Status Error
(cd0:ata1:0:0:0): SCSI status: Check Condition
(cd0:ata1:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(cd0:ata1:0:0:0): Error 6, Unretryable error
(cd0:ata1:0:0:0): cddone: got error 0x6 back
g_vfs_done():iso9660/FREEBSD_INSTALL[READ(offset=77463552,
length=2048)]error = 6
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...2 1 0 0 done
All buffers synced.
Uptime: 15m40s
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012
    root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.00-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 6  Model = 2d  Stepping = 7
  Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0x9e982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 8589934592 (8192 MB)
avail memory = 8229728256 (7848 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on
pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem
0xd8000000-0xdbffffff,0xd0800000-0xd0ffffff irq 16 at device 15.0 on
pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem
0xd0040000-0xd005ffff,0xd0020000-0xd003ffff irq 17 at device 16.0 on
pci0
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.4> port
0x2000-0x203f mem 0xd1020000-0xd103ffff,0xd1000000-0xd100ffff irq 18
at device 0.0 on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: Ethernet address: 00:0c:29:cb:87:55
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe7fff on
isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ctl: CAM Target Layer loaded
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
Timecounters tick every 10.000 msec
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI-0 device
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 12288MB (25165824 512 byte sectors: 255H 63S/T 1566C)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/da0p2 [rw]...
arp: 192.168.30.239 moved from 00:0c:29:b4:0c:79 to 00:0c:29:b4:0c:65 on em0
root at CRM-FB9:/root #