<br><div class="gmail_quote">On Tue, Dec 18, 2012 at 10:41 AM, Vasilis Liaskovitis <span dir="ltr"><<a href="mailto:vasilis.liaskovitis@profitbricks.com" target="_blank">vasilis.liaskovitis@profitbricks.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This is v4 of the ACPI memory hotplug functionality. Only x86_64 target is<br>
supported (both i440fx and q35). There are still several issues, but it's<br>
been a while since v3 and I wanted to get some more feedback on the current<br>
state of the patchseries.<br>
<br></blockquote><div><br></div><div>We are working in memory hotplug functionality on pSeries machine. I'm wondering whether and how we can better integrate things. Do you think the DIMM abstraction is generic enough to be used in other machine types?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Overview:<br>
<br>
Dimm device layout is modeled with a normal qemu device:<br>
<br>
"-device dimm,id=name,size=sz,node=pxm,populated=on|off,bus=membus.0"<br>
<br></blockquote><div> </div><div> How does this will handle the no-hotplugable memory for example the memory passed in '-m' parameter?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The starting physical address for all dimms is calculated from top of memory,<br>
during memory controller init, skipping the pci hole at [PCI_HOLE_START, 4G).<br>
e.g.<br>
"-device dimm,id=dimm0,size=512M,node=0,populated=off,bus=membus.0"<br>
will define a 512M memory dimm belonging to numa node 0, on bus membus.0.<br>
<br>
Because dimm layout needs to be configured on machine-boot, all dimm devices<br>
need to be specified on startup command line (either with populated=on or with<br>
populated=off). The dimm information is stored in dimm configuration structures.<br>
<br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">After machine startup, dimms are hot-added or removed with normal device_add<br>
and device_del operations e.g.:<br>
Hot-add syntax: "device_add dimm,id=mydimm0,bus=membus.0"<br>
Hot-remove syntax: "device_del dimm,id=mydimm0"<br>
<br>
Changes v3->v4<br>
<br>
- Dimms added with normal -device argument (extra -dimm arg dropped).<br>
- multiple memory buses can be registered. Memory buses of the real hw/chipset<br>
or a paravirtual memory bus can be added.<br>
- acpi implementation uses memory API instead of old ioports.<br>
- Support for q35/ich9 added (still buggy, see patch 12/31).<br>
- piix4/i440fx initialization code has been refactored to resemble q35. This<br>
will allow memory map initialization at chipset qdev init time for both<br>
machines, as well as more similar code.<br>
- Hot-remove functionality has been moved to separate patches. Hot-remove no<br>
longer frees memory but unmaps the dimm/qdev device from the guest's view.<br>
Freeing the memory should happen when the last user unrefs/unmaps the memory,<br>
see also (work in progress):<br>
<a href="https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg00728.html" target="_blank">https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg00728.html</a><br>
<a href="https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02697.html" target="_blank">https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02697.html</a><br>
- new qmp/hmp command for the state of each dimm (on/off)<br>
<br>
Changes v2->v3<br>
<br>
- qdev integration. Dimms are attached to a dimmbus. The dimmbus is a child<br>
of i440fx device in the pc machine. Hot-add and remove are done with normal<br>
device_add / device_del operations on the dimmbus. New commands "dimm_add" and<br>
"dimm_del" are obsolete.<br>
- Add _PS3 method to allow OSPM-induced hot operations.<br>
- pci-window calculation in Seabios takes dimms into account(for both 32-bit and<br>
64-bit windows)<br>
- rename new qmp commands: query-memory-total and query-memory-hotplug<br>
- balloon driver can see the hotplugged memory<br>
<br>
Changes v1->v2<br>
<br>
- memory map is automatically calculated for hotplug dimms. Dimms are added from<br>
top-of-memory skipping the pci hole at [PCI_HOLE_START, 4G).<br>
- Renamed from "-memslot" to "-dimm". Commands changed to "dimm_add", "dimm_del"<br>
- Seabios ejection array reduced to a byte. Use extraction macros for dimm ssdt.<br>
- additional SRAT paravirt info does not break previous SRAT fw_cfg layout.<br>
- Documentation of new acpi_piix4 registers and paravirt data.<br>
- add ACPI _OST support for _OST enabled guests. This allows qemu to receive<br>
notification for success / failure of memory hot-add and hot-remove operations.<br>
Guest needs to support _OST (<a href="https://lkml.org/lkml/2012/6/25/321" target="_blank">https://lkml.org/lkml/2012/6/25/321</a>)<br>
- add monitor info command to report total guest memory (initial + hot-added)<br>
<br>
Issues:<br>
<br>
- hot-remove needs to only unmap the dimm device from guest's view. Freeing the<br>
memory should happen when the last user of the device (e.g. virtio-blk) unrefs<br>
the device. A testcase is needed for this.<br>
<br>
- Live Migration: Ramblocks are migrated before qdev VMStates are migrated. So<br>
the DimmDevice is handled diferrently than other devices. Should this be<br>
reworked ?( DimmDevice structure currently does not define a VMStateDescription)<br>
Live migration works as long as the dimm layout (command line args) are<br>
identical at the source and destination qemu command line, and destination takes<br>
into account hot-operations that have occured on source. (v3 patch 10/19<br>
created the DimmDevice that corresponds to an unknown incoming ramblock, e.g.<br>
for a dimm that was hot-added on source. but has been dropped for the moment).<br>
<br>
- A main blocker issue is windows guest functionality. The patchset does not<br>
work for windows currently. Testing on win2012 server RC or windows2008<br>
consumer prerelease, when adding a DIMM, there is a BSOD with ACPI_BIOS_ERROR<br>
message. After this, the VM keeps rebooting with ACPI_BIOS_ERROR. The windows<br>
pnpmem driver obviosuly has a problem with the seabios dimm implementation<br>
(or the seabios dimm implementation is not fully ACPI-compliant). If someone<br>
can review the seabios patches or has any ideas to debug this, let me know.<br>
<br>
- hot-operation notification lists need to be added to migration state.<br>
<br>
series is based on:<br>
- qemu master (commit a8a826a3) + patch:<br>
<a href="https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02699.html" target="_blank">https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02699.html</a><br>
- seabios master (commit a810e4e7)<br>
<br>
Can also be found at:<br>
<br>
<a href="http://github.com/vliaskov/qemu-kvm/commits/memhp-v4" target="_blank">http://github.com/vliaskov/qemu-kvm/commits/memhp-v4</a><br>
<a href="http://github.com/vliaskov/seabios/commits/memhp-v4" target="_blank">http://github.com/vliaskov/seabios/commits/memhp-v4</a><br>
<br>
Vasilis Liaskovitis (21):<br>
qapi: make visit_type_size fallback to type_int<br>
Add SIZE type to qdev properties<br>
qemu-option: export parse_option_number<br>
Implement dimm device abstraction<br>
vl: handle "-device dimm"<br>
acpi_piix4 : Implement memory device hotplug registers<br>
acpi_ich9 : Implement memory device hotplug registers<br>
piix_pci and pc_piix: refactor<br>
piix_pci: Add i440fx dram controller initialization<br>
q35: Add i440fx dram controller initialization<br>
pc: Add dimm paravirt SRAT info<br>
Introduce paravirt interface QEMU_CFG_PCI_WINDOW<br>
Implement "info memory-total" and "query-memory-total"<br>
balloon: update with hotplugged memory<br>
Implement dimm-info<br>
dimm: add hot-remove capability<br>
acpi_piix4: add hot-remove capability<br>
acpi_ich9: add hot-remove capability<br>
Implement qmp and hmp commands for notification lists<br>
Add _OST dimm support<br>
Implement _PS3 for dimm<br>
<br>
docs/specs/acpi_hotplug.txt | 54 ++++++<br>
docs/specs/fwcfg.txt | 28 +++<br>
hmp-commands.hx | 6 +<br>
hmp.c | 41 ++++<br>
hmp.h | 3 +<br>
hw/Makefile.objs | 2 +-<br>
hw/acpi.h | 5 +<br>
hw/acpi_ich9.c | 115 +++++++++++-<br>
hw/acpi_ich9.h | 12 +-<br>
hw/acpi_piix4.c | 126 ++++++++++++-<br>
hw/dimm.c | 444 +++++++++++++++++++++++++++++++++++++++++++<br>
hw/dimm.h | 102 ++++++++++<br>
hw/fw_cfg.h | 1 +<br>
hw/lpc_ich9.c | 2 +-<br>
hw/pc.c | 28 +++-<br>
hw/pc.h | 1 +<br>
hw/pc_piix.c | 74 ++++++--<br>
hw/pc_q35.c | 18 ++-<br>
hw/piix_pci.c | 249 ++++++++-----------------<br>
hw/q35.c | 27 +++<br>
hw/q35.h | 5 +<br>
hw/qdev-properties.c | 60 ++++++<br>
hw/qdev-properties.h | 3 +<br>
hw/virtio-balloon.c | 13 +-<br>
monitor.c | 21 ++<br>
qapi-schema.json | 63 ++++++<br>
qapi/qapi-visit-core.c | 11 +-<br>
qemu-option.c | 4 +-<br>
qemu-option.h | 4 +<br>
qmp-commands.hx | 57 ++++++<br>
sysemu.h | 1 +<br>
vl.c | 60 ++++++<br>
32 files changed, 1432 insertions(+), 208 deletions(-)<br>
create mode 100644 docs/specs/acpi_hotplug.txt<br>
create mode 100644 docs/specs/fwcfg.txt<br>
create mode 100644 hw/dimm.c<br>
create mode 100644 hw/dimm.h<br>
<br>
<br>
Vasilis Liaskovitis (9):<br>
Add ACPI_EXTRACT_DEVICE* macros<br>
Add SSDT memory device support<br>
acpi-dsdt: Implement functions for memory hotplug<br>
acpi: generate hotplug memory devices<br>
q35: Add memory hotplug handler<br>
pci: Use paravirt interface for pcimem_start and pcimem64_start<br>
acpi: add _EJ0 operation and eject port for memory devices<br>
Add _OST dimm method<br>
Implement _PS3 method for memory device<br>
<br>
Makefile | 2 +-<br>
src/acpi-dsdt-mem-hotplug.dsl | 136 +++++++++++++++++++++++++++++++++++<br>
src/acpi-dsdt.dsl | 5 +-<br>
src/acpi.c | 158 +++++++++++++++++++++++++++++++++++++++--<br>
src/paravirt.c | 6 ++<br>
src/paravirt.h | 2 +<br>
src/pciinit.c | 9 +++<br>
src/q35-acpi-dsdt.dsl | 6 +-<br>
src/ssdt-mem.dsl | 73 +++++++++++++++++++<br>
tools/acpi_extract.py | 28 +++++++<br>
10 files changed, 415 insertions(+), 10 deletions(-)<br>
create mode 100644 src/acpi-dsdt-mem-hotplug.dsl<br>
create mode 100644 src/ssdt-mem.dsl<br>
<span><font color="#888888"><br>
--<br>
1.7.9<br>
<br>
<br>
</font></span></blockquote></div><br>