arm64: Port multikernel to arm64

Documentation/multikernel/arm64.rst

@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================
+Multikernel on arm64
+=====================
+
+Everything in ``usage.rst``, ``device-tree.rst`` and ``overlays.rst``
+applies unchanged. This document covers what is different below that
+interface, what a platform has to provide, and what the host and a spawn
+kernel need on their command lines.
+
+Requirements
+============
+
+- PSCI 0.2 or later (``CPU_ON``, ``CPU_OFF``, ``AFFINITY_INFO``), as any
+  SBSA server and QEMU's ``virt`` machine have. Spin-table platforms are
+  not supported.
+- A GICv3. MSIs for PCI devices of an instance need an ITS behind those
+  devices; a machine may have any number of them.
+- ``CONFIG_MULTIKERNEL=y``, which on arm64 needs ``CONFIG_KEXEC_FILE`` and
+  ``CONFIG_HOTPLUG_CPU``. An instance is loaded from an ``Image`` file with
+  ``kexec_file_load()``, not from a ``vmlinux``.
+
+The host may have booted from a device tree or from UEFI and ACPI. A
+spawn kernel always boots from a device tree.
+
+CPUs
+====
+
+Firmware holds the pool. A pool CPU is a CPU in the PSCI ``OFF`` state: it
+executes nothing, and there is no park loop or mailbox in memory. The
+host moves a CPU into the pool by offlining it, a spawn is started with
+``CPU_ON`` at its Image's entry point with the device tree address as the
+context ID, and it brings up its other CPUs with the stock PSCI enable
+method. Physical CPU IDs in the multikernel trees are MPIDR affinity
+values.
+
+A spawn kernel that halts, reboots or panics turns every one of its CPUs
+off, instead of leaving them in a WFI loop inside an image the host is
+about to overwrite. The host learns that a CPU has left from
+``AFFINITY_INFO``, and refuses to load a new image over one that has not.
+
+**Force halt** has to make each CPU of the instance turn itself off, as
+PSCI has no call to turn another CPU off. Where firmware implements SDEI,
+every kernel registers SDEI's software-signalled event 0, and the host
+first signals that event to each CPU with ``SDEI_EVENT_SIGNAL``. Firmware
+takes it at EL3 and calls the spawn's handler whatever the CPU has
+masked, all exceptions included. The handler completes the event and
+turns the CPU off. Both kernels need ``CONFIG_ARM_SDE_INTERFACE``.
+
+TF-A (2.12, and its master branch at the time of writing) breaks a CPU
+that receives event 0 while it has SDEI masked: it never ends the
+interrupt, and the CPU takes no non-secure interrupt from then on. A
+kernel masks SDEI on a CPU that is going off and around power-down idle
+states, so each CPU publishes in the instance's shared area whether it
+can take the event, and the host only signals the CPUs that say so. A
+CPU that masks SDEI between the host's check and its signal still loses
+its interrupts.
+
+The host then sends the stop IPI to every CPU of the instance, as it
+does without SDEI. A CPU that spins with interrupts masked only takes it
+as a pseudo-NMI, so a spawn kernel should also be built with
+``CONFIG_ARM64_PSEUDO_NMI`` and booted with ``irqchip.gicv3_pseudo_nmi=1``;
+``kerf load`` appends the parameter to every spawn's command line. A CPU
+that neither reaches stays out of reach, and the instance cannot be
+started again until the machine is rebooted: the host refuses to load an
+image over a CPU that is still on.
+
+The Boot Tree
+=============
+
+An arm64 kernel reads its memory, its CPUs and its interrupt controller
+from the tree in ``x0`` long before any multikernel code runs, so the
+instance tree of ``device-tree.rst`` and the boot protocol's tree are one
+and the same on arm64. The host rebuilds it on every start, from the
+instance's current resources, and adds to the instance tree:
+
+``/memory@<base>``
+    One node per range of the instance's memory.
+
+``/cpus``
+    ``enable-method = "psci"`` nodes for the instance's own CPUs, the boot
+    CPU first, followed by every other CPU of the machine. A kernel can
+    only ever online a CPU it enumerated at boot; the ones the instance
+    does not own are possible but not present until they are hot-added.
+
+``/psci``, ``/timer``, the GICv3 node
+    Copied from the host's tree, or written from the FADT, GTDT and MADT
+    on an ACPI host.
+
+``/firmware/sdei``
+    Written when the host's tree has an SDEI node or its ACPI tables an
+    SDEI table, with the host's conduit.
+
+``/chosen``
+    ``bootargs``, the initrd and the seeds from the ``kexec_file_load()``
+    call, next to the multikernel properties, and
+    ``linux,pci-probe-only``: BARs and bridge windows remain the host's to
+    assign.
+
+There is no UART node. For boot messages give the spawn an explicit
+``earlycon``, for example ``earlycon=pl011,mmio32,0x9000000 keep_bootcon``
+on QEMU's ``virt`` machine, which polls the host's UART without taking
+its interrupt.
+
+Interrupts
+==========
+
+The GIC distributor exists once and stays with the host. The GIC node of
+a boot tree carries:
+
+``multikernel,tenant``
+    The GICv3 driver leaves the distributor alone: it only checks that it
+    is enabled with affinity routing, and initialises the redistributors
+    and CPU interfaces of its own CPUs. It maps no SPIs, which fail with
+    ``-EPERM``, and sets up no LPIs of its own: the redistributors of its
+    CPUs keep the LPI tables the host enabled them with.
+
+The **doorbell** of the message ring is SGI 7. All eight non-secure SGIs
+are in use on arm64, so with ``CONFIG_MULTIKERNEL`` it takes the place of
+the KGDB roundup IPI, and KGDB falls back to its generic roundup.
+
+Devices
+=======
+
+Platform devices
+    Not supported. Their interrupts are SPIs, and SPIs live in the one
+    distributor, which stays with the host. An instance that has platform
+    devices fails to start.
+
+PCI devices
+    The root buses above an instance's devices appear as
+    ``pci-host-ecam-generic`` bridges with the devices as their only
+    children. INTx is not available. MSIs go through the host's ITSs: the
+    boot tree has a ``multikernel,gic-v3-its`` node, the spawn asks the
+    host over the message ring to map each vector, and the host answers
+    with the LPI that will arrive and the doorbell address to program into
+    the device. The host picks the ITS behind each device, so nothing about
+    the machine's ITSs appears in the tree. It drops the mappings when the
+    instance halts. A root bus with no ITS behind it gets no ``msi-map``,
+    and its devices no interrupts.
+    Unbind the host's driver before the instance is started, and bind it
+    again once the instance is stopped if the host should use the device.
+
+IOMMU
+    The SMMU stays with the host and is not described to a spawn, which
+    hands physical addresses to its devices. While an instance runs, each
+    of its PCI devices behind an SMMU is attached to a domain that maps
+    exactly the instance's memory, and the doorbell of the ITS behind that
+    device, onto itself. It
+    follows memory that is added or removed at run time. A device without
+    an IOMMU has unrestricted DMA.
+
+Testing under QEMU
+==================
+
+``qemu-system-aarch64 -M virt,gic-version=3`` exercises all of the above
+with its built-in PSCI; ``secure=on,virtualization=on`` with TF-A as the
+firmware does so against real EL3 firmware and enters the kernels at EL2;
+``iommu=smmuv3`` with ``iommu_platform=on,disable-legacy=on`` on a virtio
+PCI device puts that device behind an SMMU. A spawn needs no disk: an
+initramfs is enough.
+
+TF-A built with ``SDEI_SUPPORT=1 EL3_EXCEPTION_HANDLING=1`` provides SDEI
+on QEMU, but neither QEMU nor TF-A adds an SDEI node to the tree. Dump
+QEMU's tree with ``-machine ...,dumpdtb=virt.dtb``, add
+``firmware { sdei { compatible = "arm,sdei-1.0"; method = "smc"; }; };``
+and boot the host with ``-dtb``.

Documentation/multikernel/usage.rst

@@ -66,6 +66,13 @@ Workflow
    ``device-remove``, and destroy the instance with ``instance-remove``
    once it is stopped.
 
+Architectures
+=============
+
+x86-64 and arm64 are supported. ``arm64.rst`` describes what differs on
+arm64: how CPUs are held and started, the boot tree, interrupts, and what
+a device needs to be given to an instance.
+
 Instance States
 ===============
 

arch/arm64/Kbuild

@@ -8,6 +8,7 @@ obj-$(CONFIG_KVM)	+= kvm/
 obj-$(CONFIG_XEN)	+= xen/
 obj-$(subst m,y,$(CONFIG_HYPERV))	+= hyperv/
 obj-$(CONFIG_CRYPTO)	+= crypto/
+obj-$(CONFIG_MULTIKERNEL)	+= multikernel/
 
 # for cleaning
 subdir- += boot

arch/arm64/Kconfig

@@ -1622,6 +1622,14 @@ config ARCH_DEFAULT_KEXEC_IMAGE_VERIFY_SIG
 config ARCH_SUPPORTS_KEXEC_HANDOVER
 	def_bool y
 
+config ARCH_SUPPORTS_MULTIKERNEL
+	def_bool ARM_PSCI_FW && KEXEC_FILE && HOTPLUG_CPU
+
+# An ITS has one command queue, so the host maps MSIs for its instances
+config MULTIKERNEL_ITS_PROXY
+	def_bool MULTIKERNEL && ARM_GIC_V3_ITS && PCI_MSI
+	select ARCH_HAS_MK_MSI_PROXY
+
 config ARCH_SUPPORTS_CRASH_DUMP
 	def_bool y
 

arch/arm64/include/asm/multikernel.h

@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_MULTIKERNEL_H
+#define _ASM_ARM64_MULTIKERNEL_H
+
+#include <linux/sizes.h>
+#include <linux/types.h>
+#include <asm/page-def.h>
+#include <asm/smp_plat.h>
+
+#ifdef CONFIG_MULTIKERNEL
+/*
+ * Pool CPUs are held by firmware in the PSCI OFF state and started with
+ * CPU_ON, so an instance needs no trampoline, park page or page tables,
+ * and what CPU_ON is given lives in the kimage.
+ */
+struct mk_instance_arch {
+};
+
+struct mk_pool_arch {
+};
+#endif
+
+/* Physical CPU IDs are MPIDR affinity values, as in the logical map */
+static inline u64 arch_cpu_physical_id(int cpu)
+{
+	return cpu_logical_map(cpu);
+}
+
+static inline int arch_cpu_from_physical_id(u64 phys_id)
+{
+	return get_logical_index(phys_id);
+}
+
+/*
+ * The ways down of a spawn kernel. Stock arm64 leaves stopped CPUs in a
+ * WFI loop inside the kernel image, which the host is about to overwrite;
+ * a spawn returns them to firmware instead. Both return on a host.
+ */
+#ifdef CONFIG_MULTIKERNEL
+void mk_spawn_machine_halt(void);
+void mk_spawn_stop_this_cpu(void);
+
+/* A stop IPI from another kernel: its force halt, or not ours to obey */
+void mk_foreign_cpu_stop(void);
+
+/* The boot tree is the manifest when another kernel wrote it */
+void mk_spawn_accept_boot_tree(phys_addr_t dt_phys);
+#else
+static inline void mk_spawn_accept_boot_tree(phys_addr_t dt_phys)
+{
+}
+
+static inline void mk_spawn_machine_halt(void)
+{
+}
+
+static inline void mk_spawn_stop_this_cpu(void)
+{
+}
+
+static inline void mk_foreign_cpu_stop(void)
+{
+}
+#endif
+
+/*
+ * Ask the host to route a device's MSI to one of this kernel's CPUs, and to
+ * move it to another later, for the instance's MSI domain
+ * (irq-gic-v3-its-mk.c). Mapping sleeps; moving works in any context and
+ * is done when it returns.
+ */
+#ifdef CONFIG_MULTIKERNEL_ITS_PROXY
+int mk_msi_proxy_map(u32 domain, u32 rid, u32 event, u64 phys_cpu,
+		     phys_addr_t *doorbell);
+int mk_msi_proxy_move(u32 domain, u32 rid, u32 event, u64 phys_cpu);
+#endif
+
+/*
+ * Room for an instance's boot device tree, a segment of its kimage: the
+ * manifest (MK_MANIFEST_SIZE) with the arm64 boot nodes on top. The tree
+ * is rebuilt in place on every spawn, as the memory grant can have
+ * changed since the load.
+ */
+#define MK_BOOT_DTB_SIZE	SZ_512K
+
+/* Nothing of arm64's lives in the control block, so it is never allocated */
+#define MK_CTRL_BLOCK_SIZE	PAGE_SIZE
+
+#endif /* _ASM_ARM64_MULTIKERNEL_H */

arch/arm64/include/asm/smp.h

@@ -64,6 +64,14 @@ enum ipi_msg_type {
 	 */
 	IPI_CPU_BACKTRACE = NR_IPI,
 	IPI_KGDB_ROUNDUP,
+	/*
+	 * The multikernel doorbell goes to CPUs of other kernels, which
+	 * makes its SGI number ABI between them. All eight non-secure SGIs
+	 * are taken, so with CONFIG_MULTIKERNEL it has the one of the KGDB
+	 * roundup, which can do without: the generic roundup needs no IPI
+	 * of its own.
+	 */
+	IPI_MULTIKERNEL = IPI_KGDB_ROUNDUP,
 	MAX_IPI
 };
 

arch/arm64/kernel/machine_kexec.c

@@ -54,7 +54,7 @@ void machine_kexec_cleanup(struct kimage *kimage)
  */
 int machine_kexec_prepare(struct kimage *kimage)
 {
-	if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) {
+	if (kimage->type == KEXEC_TYPE_DEFAULT && cpus_are_stuck_in_kernel()) {
 		pr_err("Can't kexec: CPUs are stuck in the kernel.\n");
 		return -EBUSY;
 	}
@@ -113,6 +113,14 @@ int machine_kexec_post_load(struct kimage *kimage)
 		.trans_alloc_arg	= kimage,
 	};
 
+	/*
+	 * A multikernel image is loaded in place and never replaces this
+	 * kernel: no relocation code, no copy of the linear map. It is
+	 * copied in again and flushed on every spawn, not here.
+	 */
+	if (kimage->type == KEXEC_TYPE_MULTIKERNEL)
+		return 0;
+
 	/* If in place, relocation is not used, only flush next kernel */
 	if (kimage->head & IND_DONE) {
 		kexec_segment_flush(kimage);

arch/arm64/kernel/machine_kexec_file.c

@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/types.h>
 #include <linux/vmalloc.h>
+#include <asm/multikernel.h>
 
 const struct kexec_file_ops * const kexec_file_loaders[] = {
 	&kexec_image_ops,
@@ -174,6 +175,12 @@ int load_other_segments(struct kimage *image,
 	kbuf.bufsz = dtb_len;
 	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
 	kbuf.memsz = dtb_len;
+	/*
+	 * A multikernel instance does not boot from this tree but from one
+	 * rebuilt in its place on every spawn. Leave room for that.
+	 */
+	if (image->type == KEXEC_TYPE_MULTIKERNEL)
+		kbuf.memsz = max(dtb_len, MK_BOOT_DTB_SIZE);
 	/* not across 2MB boundary */
 	kbuf.buf_align = SZ_2M;
 	kbuf.buf_max = ULONG_MAX;

arch/arm64/kernel/process.c

@@ -52,6 +52,7 @@
 #include <asm/gcs.h>
 #include <asm/mmu_context.h>
 #include <asm/mte.h>
+#include <asm/multikernel.h>
 #include <asm/processor.h>
 #include <asm/pointer_auth.h>
 #include <asm/stacktrace.h>
@@ -98,6 +99,7 @@ void machine_shutdown(void)
  */
 void machine_halt(void)
 {
+	mk_spawn_machine_halt();
 	local_irq_disable();
 	smp_send_stop();
 	while (1);
@@ -111,6 +113,7 @@ void machine_halt(void)
  */
 void machine_power_off(void)
 {
+	mk_spawn_machine_halt();
 	local_irq_disable();
 	smp_send_stop();
 	do_kernel_power_off();
@@ -127,6 +130,8 @@ void machine_power_off(void)
  */
 void machine_restart(char *cmd)
 {
+	mk_spawn_machine_halt();
+
 	/* Disable interrupts first */
 	local_irq_disable();
 	smp_send_stop();