diff --git a/Documentation/multikernel/arm64.rst b/Documentation/multikernel/arm64.rst new file mode 100644 index 00000000000000..53b96425422f56 --- /dev/null +++ b/Documentation/multikernel/arm64.rst @@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================== +Multikernel on arm64 +===================== + +Everything in ``usage.rst``, ``device-tree.rst`` and ``overlays.rst`` +applies unchanged. This document covers what is different below that +interface, what a platform has to provide, and what the host and a spawn +kernel need on their command lines. + +Requirements +============ + +- PSCI 0.2 or later (``CPU_ON``, ``CPU_OFF``, ``AFFINITY_INFO``), as any + SBSA server and QEMU's ``virt`` machine have. Spin-table platforms are + not supported. +- A GICv3. MSIs for PCI devices of an instance need an ITS behind those + devices; a machine may have any number of them. +- ``CONFIG_MULTIKERNEL=y``, which on arm64 needs ``CONFIG_KEXEC_FILE`` and + ``CONFIG_HOTPLUG_CPU``. An instance is loaded from an ``Image`` file with + ``kexec_file_load()``, not from a ``vmlinux``. + +The host may have booted from a device tree or from UEFI and ACPI. A +spawn kernel always boots from a device tree. + +CPUs +==== + +Firmware holds the pool. A pool CPU is a CPU in the PSCI ``OFF`` state: it +executes nothing, and there is no park loop or mailbox in memory. The +host moves a CPU into the pool by offlining it, a spawn is started with +``CPU_ON`` at its Image's entry point with the device tree address as the +context ID, and it brings up its other CPUs with the stock PSCI enable +method. Physical CPU IDs in the multikernel trees are MPIDR affinity +values. + +A spawn kernel that halts, reboots or panics turns every one of its CPUs +off, instead of leaving them in a WFI loop inside an image the host is +about to overwrite. The host learns that a CPU has left from +``AFFINITY_INFO``, and refuses to load a new image over one that has not. + +**Force halt** has to make each CPU of the instance turn itself off, as +PSCI has no call to turn another CPU off. Where firmware implements SDEI, +every kernel registers SDEI's software-signalled event 0, and the host +first signals that event to each CPU with ``SDEI_EVENT_SIGNAL``. Firmware +takes it at EL3 and calls the spawn's handler whatever the CPU has +masked, all exceptions included. The handler completes the event and +turns the CPU off. Both kernels need ``CONFIG_ARM_SDE_INTERFACE``. + +TF-A (2.12, and its master branch at the time of writing) breaks a CPU +that receives event 0 while it has SDEI masked: it never ends the +interrupt, and the CPU takes no non-secure interrupt from then on. A +kernel masks SDEI on a CPU that is going off and around power-down idle +states, so each CPU publishes in the instance's shared area whether it +can take the event, and the host only signals the CPUs that say so. A +CPU that masks SDEI between the host's check and its signal still loses +its interrupts. + +The host then sends the stop IPI to every CPU of the instance, as it +does without SDEI. A CPU that spins with interrupts masked only takes it +as a pseudo-NMI, so a spawn kernel should also be built with +``CONFIG_ARM64_PSEUDO_NMI`` and booted with ``irqchip.gicv3_pseudo_nmi=1``; +``kerf load`` appends the parameter to every spawn's command line. A CPU +that neither reaches stays out of reach, and the instance cannot be +started again until the machine is rebooted: the host refuses to load an +image over a CPU that is still on. + +The Boot Tree +============= + +An arm64 kernel reads its memory, its CPUs and its interrupt controller +from the tree in ``x0`` long before any multikernel code runs, so the +instance tree of ``device-tree.rst`` and the boot protocol's tree are one +and the same on arm64. The host rebuilds it on every start, from the +instance's current resources, and adds to the instance tree: + +``/memory@`` + One node per range of the instance's memory. + +``/cpus`` + ``enable-method = "psci"`` nodes for the instance's own CPUs, the boot + CPU first, followed by every other CPU of the machine. A kernel can + only ever online a CPU it enumerated at boot; the ones the instance + does not own are possible but not present until they are hot-added. + +``/psci``, ``/timer``, the GICv3 node + Copied from the host's tree, or written from the FADT, GTDT and MADT + on an ACPI host. + +``/firmware/sdei`` + Written when the host's tree has an SDEI node or its ACPI tables an + SDEI table, with the host's conduit. + +``/chosen`` + ``bootargs``, the initrd and the seeds from the ``kexec_file_load()`` + call, next to the multikernel properties, and + ``linux,pci-probe-only``: BARs and bridge windows remain the host's to + assign. + +There is no UART node. For boot messages give the spawn an explicit +``earlycon``, for example ``earlycon=pl011,mmio32,0x9000000 keep_bootcon`` +on QEMU's ``virt`` machine, which polls the host's UART without taking +its interrupt. + +Interrupts +========== + +The GIC distributor exists once and stays with the host. The GIC node of +a boot tree carries: + +``multikernel,tenant`` + The GICv3 driver leaves the distributor alone: it only checks that it + is enabled with affinity routing, and initialises the redistributors + and CPU interfaces of its own CPUs. It maps no SPIs, which fail with + ``-EPERM``, and sets up no LPIs of its own: the redistributors of its + CPUs keep the LPI tables the host enabled them with. + +The **doorbell** of the message ring is SGI 7. All eight non-secure SGIs +are in use on arm64, so with ``CONFIG_MULTIKERNEL`` it takes the place of +the KGDB roundup IPI, and KGDB falls back to its generic roundup. + +Devices +======= + +Platform devices + Not supported. Their interrupts are SPIs, and SPIs live in the one + distributor, which stays with the host. An instance that has platform + devices fails to start. + +PCI devices + The root buses above an instance's devices appear as + ``pci-host-ecam-generic`` bridges with the devices as their only + children. INTx is not available. MSIs go through the host's ITSs: the + boot tree has a ``multikernel,gic-v3-its`` node, the spawn asks the + host over the message ring to map each vector, and the host answers + with the LPI that will arrive and the doorbell address to program into + the device. The host picks the ITS behind each device, so nothing about + the machine's ITSs appears in the tree. It drops the mappings when the + instance halts. A root bus with no ITS behind it gets no ``msi-map``, + and its devices no interrupts. + Unbind the host's driver before the instance is started, and bind it + again once the instance is stopped if the host should use the device. + +IOMMU + The SMMU stays with the host and is not described to a spawn, which + hands physical addresses to its devices. While an instance runs, each + of its PCI devices behind an SMMU is attached to a domain that maps + exactly the instance's memory, and the doorbell of the ITS behind that + device, onto itself. It + follows memory that is added or removed at run time. A device without + an IOMMU has unrestricted DMA. + +Testing under QEMU +================== + +``qemu-system-aarch64 -M virt,gic-version=3`` exercises all of the above +with its built-in PSCI; ``secure=on,virtualization=on`` with TF-A as the +firmware does so against real EL3 firmware and enters the kernels at EL2; +``iommu=smmuv3`` with ``iommu_platform=on,disable-legacy=on`` on a virtio +PCI device puts that device behind an SMMU. A spawn needs no disk: an +initramfs is enough. + +TF-A built with ``SDEI_SUPPORT=1 EL3_EXCEPTION_HANDLING=1`` provides SDEI +on QEMU, but neither QEMU nor TF-A adds an SDEI node to the tree. Dump +QEMU's tree with ``-machine ...,dumpdtb=virt.dtb``, add +``firmware { sdei { compatible = "arm,sdei-1.0"; method = "smc"; }; };`` +and boot the host with ``-dtb``. diff --git a/Documentation/multikernel/usage.rst b/Documentation/multikernel/usage.rst index 8ed85d8ead7e18..7a5749ffa56fa7 100644 --- a/Documentation/multikernel/usage.rst +++ b/Documentation/multikernel/usage.rst @@ -66,6 +66,13 @@ Workflow ``device-remove``, and destroy the instance with ``instance-remove`` once it is stopped. +Architectures +============= + +x86-64 and arm64 are supported. ``arm64.rst`` describes what differs on +arm64: how CPUs are held and started, the boot tree, interrupts, and what +a device needs to be given to an instance. + Instance States =============== diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild index d876bc0e542110..39e3e68cf3d686 100644 --- a/arch/arm64/Kbuild +++ b/arch/arm64/Kbuild @@ -8,6 +8,7 @@ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/ obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/ obj-$(CONFIG_CRYPTO) += crypto/ +obj-$(CONFIG_MULTIKERNEL) += multikernel/ # for cleaning subdir- += boot diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9ea19b74b6c3c9..a1f9604580aeba 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1622,6 +1622,14 @@ config ARCH_DEFAULT_KEXEC_IMAGE_VERIFY_SIG config ARCH_SUPPORTS_KEXEC_HANDOVER def_bool y +config ARCH_SUPPORTS_MULTIKERNEL + def_bool ARM_PSCI_FW && KEXEC_FILE && HOTPLUG_CPU + +# An ITS has one command queue, so the host maps MSIs for its instances +config MULTIKERNEL_ITS_PROXY + def_bool MULTIKERNEL && ARM_GIC_V3_ITS && PCI_MSI + select ARCH_HAS_MK_MSI_PROXY + config ARCH_SUPPORTS_CRASH_DUMP def_bool y diff --git a/arch/arm64/include/asm/multikernel.h b/arch/arm64/include/asm/multikernel.h new file mode 100644 index 00000000000000..382d37a7125827 --- /dev/null +++ b/arch/arm64/include/asm/multikernel.h @@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_MULTIKERNEL_H +#define _ASM_ARM64_MULTIKERNEL_H + +#include +#include +#include +#include + +#ifdef CONFIG_MULTIKERNEL +/* + * Pool CPUs are held by firmware in the PSCI OFF state and started with + * CPU_ON, so an instance needs no trampoline, park page or page tables, + * and what CPU_ON is given lives in the kimage. + */ +struct mk_instance_arch { +}; + +struct mk_pool_arch { +}; +#endif + +/* Physical CPU IDs are MPIDR affinity values, as in the logical map */ +static inline u64 arch_cpu_physical_id(int cpu) +{ + return cpu_logical_map(cpu); +} + +static inline int arch_cpu_from_physical_id(u64 phys_id) +{ + return get_logical_index(phys_id); +} + +/* + * The ways down of a spawn kernel. Stock arm64 leaves stopped CPUs in a + * WFI loop inside the kernel image, which the host is about to overwrite; + * a spawn returns them to firmware instead. Both return on a host. + */ +#ifdef CONFIG_MULTIKERNEL +void mk_spawn_machine_halt(void); +void mk_spawn_stop_this_cpu(void); + +/* A stop IPI from another kernel: its force halt, or not ours to obey */ +void mk_foreign_cpu_stop(void); + +/* The boot tree is the manifest when another kernel wrote it */ +void mk_spawn_accept_boot_tree(phys_addr_t dt_phys); +#else +static inline void mk_spawn_accept_boot_tree(phys_addr_t dt_phys) +{ +} + +static inline void mk_spawn_machine_halt(void) +{ +} + +static inline void mk_spawn_stop_this_cpu(void) +{ +} + +static inline void mk_foreign_cpu_stop(void) +{ +} +#endif + +/* + * Ask the host to route a device's MSI to one of this kernel's CPUs, and to + * move it to another later, for the instance's MSI domain + * (irq-gic-v3-its-mk.c). Mapping sleeps; moving works in any context and + * is done when it returns. + */ +#ifdef CONFIG_MULTIKERNEL_ITS_PROXY +int mk_msi_proxy_map(u32 domain, u32 rid, u32 event, u64 phys_cpu, + phys_addr_t *doorbell); +int mk_msi_proxy_move(u32 domain, u32 rid, u32 event, u64 phys_cpu); +#endif + +/* + * Room for an instance's boot device tree, a segment of its kimage: the + * manifest (MK_MANIFEST_SIZE) with the arm64 boot nodes on top. The tree + * is rebuilt in place on every spawn, as the memory grant can have + * changed since the load. + */ +#define MK_BOOT_DTB_SIZE SZ_512K + +/* Nothing of arm64's lives in the control block, so it is never allocated */ +#define MK_CTRL_BLOCK_SIZE PAGE_SIZE + +#endif /* _ASM_ARM64_MULTIKERNEL_H */ diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 10ea4f54306903..4a3621b846d2e5 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -64,6 +64,14 @@ enum ipi_msg_type { */ IPI_CPU_BACKTRACE = NR_IPI, IPI_KGDB_ROUNDUP, + /* + * The multikernel doorbell goes to CPUs of other kernels, which + * makes its SGI number ABI between them. All eight non-secure SGIs + * are taken, so with CONFIG_MULTIKERNEL it has the one of the KGDB + * roundup, which can do without: the generic roundup needs no IPI + * of its own. + */ + IPI_MULTIKERNEL = IPI_KGDB_ROUNDUP, MAX_IPI }; diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 239c16e3d02f29..9362268755252d 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -54,7 +54,7 @@ void machine_kexec_cleanup(struct kimage *kimage) */ int machine_kexec_prepare(struct kimage *kimage) { - if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) { + if (kimage->type == KEXEC_TYPE_DEFAULT && cpus_are_stuck_in_kernel()) { pr_err("Can't kexec: CPUs are stuck in the kernel.\n"); return -EBUSY; } @@ -113,6 +113,14 @@ int machine_kexec_post_load(struct kimage *kimage) .trans_alloc_arg = kimage, }; + /* + * A multikernel image is loaded in place and never replaces this + * kernel: no relocation code, no copy of the linear map. It is + * copied in again and flushed on every spawn, not here. + */ + if (kimage->type == KEXEC_TYPE_MULTIKERNEL) + return 0; + /* If in place, relocation is not used, only flush next kernel */ if (kimage->head & IND_DONE) { kexec_segment_flush(kimage); diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c index fba260ad87a967..a499cd747e4dc9 100644 --- a/arch/arm64/kernel/machine_kexec_file.c +++ b/arch/arm64/kernel/machine_kexec_file.c @@ -21,6 +21,7 @@ #include #include #include +#include const struct kexec_file_ops * const kexec_file_loaders[] = { &kexec_image_ops, @@ -174,6 +175,12 @@ int load_other_segments(struct kimage *image, kbuf.bufsz = dtb_len; kbuf.mem = KEXEC_BUF_MEM_UNKNOWN; kbuf.memsz = dtb_len; + /* + * A multikernel instance does not boot from this tree but from one + * rebuilt in its place on every spawn. Leave room for that. + */ + if (image->type == KEXEC_TYPE_MULTIKERNEL) + kbuf.memsz = max(dtb_len, MK_BOOT_DTB_SIZE); /* not across 2MB boundary */ kbuf.buf_align = SZ_2M; kbuf.buf_max = ULONG_MAX; diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 489554931231e6..20a7d00ac8d90b 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -98,6 +99,7 @@ void machine_shutdown(void) */ void machine_halt(void) { + mk_spawn_machine_halt(); local_irq_disable(); smp_send_stop(); while (1); @@ -111,6 +113,7 @@ void machine_halt(void) */ void machine_power_off(void) { + mk_spawn_machine_halt(); local_irq_disable(); smp_send_stop(); do_kernel_power_off(); @@ -127,6 +130,8 @@ void machine_power_off(void) */ void machine_restart(char *cmd) { + mk_spawn_machine_halt(); + /* Disable interrupts first */ local_irq_disable(); smp_send_stop(); diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 23c05dc7a8f2ac..ec7960a86c5e9e 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -54,6 +54,7 @@ #include #include #include +#include static int num_standard_resources; static struct resource *standard_resources; @@ -290,6 +291,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) early_ioremap_init(); setup_machine_fdt(__fdt_pointer); + mk_spawn_accept_boot_tree(__fdt_pointer); /* * Initialise the static keys early as they may be enabled by the diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 1aa324104afb44..c4a7ee9f5ef62d 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -45,6 +46,7 @@ #include #include #include +#include #include #include #include @@ -78,6 +80,8 @@ static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc); static bool percpu_ipi_descs __ro_after_init; static bool crash_stop; +/* Set once this kernel starts stopping its own CPUs */ +static unsigned long stop_in_progress; static void ipi_setup(int cpu); @@ -821,7 +825,11 @@ static const char *ipi_types[MAX_IPI] __tracepoint_string = { [IPI_TIMER] = "Timer broadcast interrupts", [IPI_IRQ_WORK] = "IRQ work interrupts", [IPI_CPU_BACKTRACE] = "CPU backtrace interrupts", +#ifdef CONFIG_MULTIKERNEL + [IPI_MULTIKERNEL] = "Multikernel doorbell interrupts", +#else [IPI_KGDB_ROUNDUP] = "KGDB roundup interrupts", +#endif }; static void smp_cross_call(const struct cpumask *target, unsigned int ipinr); @@ -866,6 +874,7 @@ static void __noreturn local_cpu_stop(unsigned int cpu) set_cpu_online(cpu, false); local_daif_mask(); + mk_spawn_stop_this_cpu(); sdei_mask_local_cpu(); cpu_park_loop(); } @@ -936,7 +945,7 @@ void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu) nmi_trigger_cpumask_backtrace(mask, exclude_cpu, arm64_backtrace_ipi); } -#ifdef CONFIG_KGDB +#if defined(CONFIG_KGDB) && !defined(CONFIG_MULTIKERNEL) void kgdb_roundup_cpus(void) { int this_cpu = raw_smp_processor_id(); @@ -952,6 +961,16 @@ void kgdb_roundup_cpus(void) } #endif +/* + * SGIs cross kernel boundaries, so with multikernel a stop IPI that this + * kernel did not send itself comes from another kernel. + */ +static bool ipi_cpu_stop_is_foreign(void) +{ + return IS_ENABLED(CONFIG_MULTIKERNEL) && + !READ_ONCE(stop_in_progress) && !crash_stop; +} + /* * Main handler for inter-processor interrupts */ @@ -973,6 +992,10 @@ static void do_handle_IPI(int ipinr) case IPI_CPU_STOP: case IPI_CPU_STOP_NMI: + if (ipi_cpu_stop_is_foreign()) { + mk_foreign_cpu_stop(); + break; + } if (IS_ENABLED(CONFIG_KEXEC_CORE) && crash_stop) { ipi_cpu_crash_stop(cpu, get_irq_regs()); unreachable(); @@ -1002,7 +1025,10 @@ static void do_handle_IPI(int ipinr) break; case IPI_KGDB_ROUNDUP: - kgdb_nmicallback(cpu, get_irq_regs()); + if (IS_ENABLED(CONFIG_MULTIKERNEL)) + generic_multikernel_interrupt(); + else + kgdb_nmicallback(cpu, get_irq_regs()); break; default: @@ -1036,8 +1062,10 @@ static bool ipi_should_be_nmi(enum ipi_msg_type ipi) switch (ipi) { case IPI_CPU_STOP_NMI: case IPI_CPU_BACKTRACE: - case IPI_KGDB_ROUNDUP: return true; + case IPI_KGDB_ROUNDUP: + /* The doorbell drains a message ring, which is no NMI work */ + return !IS_ENABLED(CONFIG_MULTIKERNEL); default: return false; } @@ -1184,7 +1212,6 @@ static inline unsigned int num_other_online_cpus(void) void smp_send_stop(void) { - static unsigned long stop_in_progress; static cpumask_t mask; unsigned long timeout; diff --git a/arch/arm64/multikernel/Makefile b/arch/arm64/multikernel/Makefile new file mode 100644 index 00000000000000..ddf0495610e5ce --- /dev/null +++ b/arch/arm64/multikernel/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for multikernel spawn support +# + +obj-y += spawn.o dtb.o dtb_of.o dtb_pci.o +obj-$(CONFIG_ACPI) += dtb_acpi.o +obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o +obj-$(CONFIG_MULTIKERNEL_ITS_PROXY) += its_proxy.o diff --git a/arch/arm64/multikernel/dtb.c b/arch/arm64/multikernel/dtb.c new file mode 100644 index 00000000000000..7e2441ff188268 --- /dev/null +++ b/arch/arm64/multikernel/dtb.c @@ -0,0 +1,250 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * The device tree an arm64 instance boots from. + * + * On arm64 the tree in x0 is the only boot channel, so it carries both + * halves of what x86 splits between boot_params and the manifest: the + * instance tree the generic code wrote, with its /resources and /chosen, + * is the base, and the nodes the arm64 boot path itself reads go on top: + * /memory, /cpus, PSCI, the arch timer, the interrupt controller and SDEI. + * + * It is rebuilt on every spawn, into the device tree segment the loader + * left room for: the memory grant and the CPUs can change between load + * and exec, and a tree made at load would hand the spawn what it owned + * back then. + */ + +#define pr_fmt(fmt) "mk_dtb: " fmt + +#include +#include +#include +#include +#include + +#include "internal.h" + +static_assert(MK_MANIFEST_SIZE + SZ_64K <= MK_BOOT_DTB_SIZE); + +/* What the loader's tree knows and nobody else does */ +static const char * const mk_chosen_props[] = { + "bootargs", + "linux,initrd-start", + "linux,initrd-end", + "kaslr-seed", + "rng-seed", +}; + +/* + * Every wired interrupt goes through the one GIC distributor, which stays + * the host's, so an instance has no platform devices: its devices are PCI + * functions with MSIs. The spawn kernel's GIC driver leaves the + * distributor alone and maps no SPIs. + */ +static int mk_dtb_add_tenancy(void *fdt, struct mk_instance *instance) +{ + int node; + + if (instance->platform_devices_valid && + !list_empty(&instance->platform_devices)) { + pr_err("Instance %d: platform devices cannot be given to an instance on arm64\n", + instance->id); + return -FDT_ERR_BADSTATE; + } + + node = fdt_node_offset_by_compatible(fdt, -1, "arm,gic-v3"); + if (node < 0) + return node; + return fdt_setprop_empty(fdt, node, "multikernel,tenant"); +} + +/* @pairs are base,size; the root of the instance tree has two cells for each */ +int mk_dtb_set_reg(void *fdt, int node, const u64 *pairs, int nr) +{ + int i, ret; + + ret = fdt_delprop(fdt, node, "reg"); + if (ret && ret != -FDT_ERR_NOTFOUND) + return ret; + + for (i = 0; i < 2 * nr; i++) { + ret = fdt_appendprop_u64(fdt, node, "reg", pairs[i]); + if (ret) + return ret; + } + return 0; +} + +static int mk_dtb_add_memory(void *fdt, struct mk_instance *instance) +{ + struct mk_memory_region *region; + char name[32]; + int node, ret; + + list_for_each_entry(region, &instance->memory_regions, list) { + u64 reg[2] = { region->res.start, resource_size(®ion->res) }; + + snprintf(name, sizeof(name), "memory@%llx", reg[0]); + node = fdt_add_subnode(fdt, 0, name); + if (node < 0) + return node; + ret = fdt_setprop_string(fdt, node, "device_type", "memory"); + if (!ret) + ret = mk_dtb_set_reg(fdt, node, reg, 1); + if (ret) + return ret; + } + return 0; +} + +static int mk_dtb_add_cpu(void *fdt, int cpus, mk_phys_cpu_t mpidr) +{ + char name[32]; + int node, ret; + + snprintf(name, sizeof(name), "cpu@%llx", mpidr); + node = fdt_add_subnode(fdt, cpus, name); + if (node < 0) + return node; + + ret = fdt_setprop_string(fdt, node, "device_type", "cpu"); + if (!ret) + ret = fdt_setprop_string(fdt, node, "compatible", "arm,armv8"); + if (!ret) + ret = fdt_setprop_string(fdt, node, "enable-method", "psci"); + if (!ret) + ret = fdt_setprop_u64(fdt, node, "reg", mpidr); + return ret; +} + +/* + * The instance's own CPUs come first, the boot CPU leading as it does in + * instance->cpus, so they get the low logical numbers. Every other CPU + * of the machine follows: a kernel can only ever online a CPU it + * enumerated at boot, and the generic code prunes the ones the instance + * does not own from the present mask. + * + * fdt_add_subnode() puts a new node ahead of its siblings, hence the + * walk from the last CPU to the first. + */ +static int mk_dtb_add_cpus(void *fdt, struct mk_instance *instance) +{ + mk_phys_cpu_t mpidr; + unsigned int i; + int cpus, cpu, ret; + + cpus = fdt_add_subnode(fdt, 0, "cpus"); + if (cpus < 0) + return cpus; + ret = fdt_setprop_u32(fdt, cpus, "#address-cells", 2); + if (!ret) + ret = fdt_setprop_u32(fdt, cpus, "#size-cells", 0); + if (ret) + return ret; + + for (cpu = nr_cpu_ids - 1; cpu >= 0; cpu--) { + if (!cpu_possible(cpu)) + continue; + mpidr = arch_cpu_physical_id(cpu); + if (mpidr == INVALID_HWID || + mk_cpu_set_contains(instance->cpus, mpidr)) + continue; + ret = mk_dtb_add_cpu(fdt, cpus, mpidr); + if (ret) + return ret; + } + + for (i = mk_cpu_set_count(instance->cpus); i-- > 0; ) { + ret = mk_dtb_add_cpu(fdt, cpus, instance->cpus->ids[i]); + if (ret) + return ret; + } + return 0; +} + +static int mk_dtb_add_chosen(void *fdt, const void *loader_fdt) +{ + int from, to, i; + + from = fdt_path_offset(loader_fdt, "/chosen"); + if (from < 0) + return 0; + + to = fdt_path_offset(fdt, "/chosen"); + if (to < 0) + return to; + + for (i = 0; i < ARRAY_SIZE(mk_chosen_props); i++) { + const void *val; + int len, ret; + + val = fdt_getprop(loader_fdt, from, mk_chosen_props[i], &len); + if (!val) + continue; + ret = fdt_setprop(fdt, to, mk_chosen_props[i], val, len); + if (ret) + return ret; + } + + /* BARs and bridge windows are the host's to assign, for every kernel */ + return fdt_setprop_u32(fdt, to, "linux,pci-probe-only", 1); +} + +static struct kexec_segment *mk_dtb_segment(struct kimage *image) +{ + unsigned long i; + + for (i = 0; i < image->nr_segments; i++) { + if (image->segment[i].mem == image->arch.dtb_mem) + return &image->segment[i]; + } + return NULL; +} + +/** + * mk_build_boot_dtb - Write an instance's boot tree over the loader's + * @image: Loaded kimage, its manifest already finalized for this spawn + * @instance: Instance being spawned + * + * Everything that adds to the tree returns libfdt error codes, which + * share their values with errnos and must not be mixed with them. + * + * Returns 0, -ENOSPC when the segment is too small, or -EINVAL. + */ +int mk_build_boot_dtb(struct kimage *image, struct mk_instance *instance) +{ + struct kexec_segment *seg = mk_dtb_segment(image); + void *fdt; + int ret; + + if (!seg || !image->arch.dtb || !instance->manifest_phys) + return -EINVAL; + + fdt = phys_to_virt(seg->mem); + ret = fdt_open_into(phys_to_virt(instance->manifest_phys), fdt, seg->memsz); + if (!ret) + ret = mk_dtb_add_memory(fdt, instance); + if (!ret) + ret = mk_dtb_add_cpus(fdt, instance); + if (!ret) + ret = mk_dtb_add_chosen(fdt, image->arch.dtb); + if (!ret) + ret = acpi_disabled ? mk_dtb_add_platform_of(fdt) : + mk_dtb_add_platform_acpi(fdt); + if (!ret) + ret = mk_dtb_add_sdei(fdt); + if (!ret) + ret = mk_dtb_add_tenancy(fdt, instance); + if (!ret) + ret = mk_dtb_add_pci(fdt); + if (!ret) + ret = fdt_pack(fdt); + if (ret) { + pr_err("Instance %d: %s\n", instance->id, fdt_strerror(ret)); + return ret == -FDT_ERR_NOSPACE ? -ENOSPC : -EINVAL; + } + + pr_info("Instance %d boots from a %u byte tree at %#lx\n", + instance->id, fdt_totalsize(fdt), seg->mem); + return 0; +} diff --git a/arch/arm64/multikernel/dtb_acpi.c b/arch/arm64/multikernel/dtb_acpi.c new file mode 100644 index 00000000000000..6e4c7ab28b95a0 --- /dev/null +++ b/arch/arm64/multikernel/dtb_acpi.c @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Platform nodes of an instance tree, on a host that booted with ACPI. + * + * A spawn kernel always boots from a device tree, so what the host read + * from its tables is written out in the generic bindings: PSCI's conduit + * from the FADT, the arch timer's PPIs from the GTDT, and the GICv3 + * frames from the MADT. SBSA guarantees the generic drivers bind to it. + */ + +#define pr_fmt(fmt) "mk_dtb: " fmt + +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +#define MK_GICR_MAX_REGIONS 256 + +static int mk_dtb_add_psci(void *fdt) +{ + static const char compatible[] = "arm,psci-1.0\0arm,psci-0.2"; + int node, ret; + + node = fdt_add_subnode(fdt, 0, "psci"); + if (node < 0) + return node; + + ret = fdt_setprop(fdt, node, "compatible", compatible, sizeof(compatible)); + if (!ret) + ret = fdt_setprop_string(fdt, node, "method", + acpi_psci_use_hvc() ? "hvc" : "smc"); + return ret; +} + +static int mk_dtb_append_ppi(void *fdt, int node, u32 gsiv, u32 gtdt_flags) +{ + bool edge = gtdt_flags & ACPI_GTDT_INTERRUPT_MODE; + bool low = gtdt_flags & ACPI_GTDT_INTERRUPT_POLARITY; + u32 type; + int ret; + + if (edge) + type = low ? IRQ_TYPE_EDGE_FALLING : IRQ_TYPE_EDGE_RISING; + else + type = low ? IRQ_TYPE_LEVEL_LOW : IRQ_TYPE_LEVEL_HIGH; + + ret = fdt_appendprop_u32(fdt, node, "interrupts", MK_GIC_PPI); + if (!ret) + ret = fdt_appendprop_u32(fdt, node, "interrupts", gsiv - 16); + if (!ret) + ret = fdt_appendprop_u32(fdt, node, "interrupts", type); + return ret; +} + +/* The binding's order: secure, non-secure, virtual, hypervisor */ +static int mk_dtb_add_timer(void *fdt) +{ + struct acpi_table_header *table; + struct acpi_table_gtdt *gtdt; + int node, ret; + + if (ACPI_FAILURE(acpi_get_table(ACPI_SIG_GTDT, 0, &table))) { + pr_err("No GTDT to describe the arch timer from\n"); + return -FDT_ERR_NOTFOUND; + } + gtdt = (struct acpi_table_gtdt *)table; + + node = fdt_add_subnode(fdt, 0, "timer"); + ret = node < 0 ? node : 0; + if (!ret) + ret = fdt_setprop_string(fdt, node, "compatible", "arm,armv8-timer"); + if (!ret) + ret = mk_dtb_append_ppi(fdt, node, gtdt->secure_el1_interrupt, + gtdt->secure_el1_flags); + if (!ret) + ret = mk_dtb_append_ppi(fdt, node, gtdt->non_secure_el1_interrupt, + gtdt->non_secure_el1_flags); + if (!ret) + ret = mk_dtb_append_ppi(fdt, node, gtdt->virtual_timer_interrupt, + gtdt->virtual_timer_flags); + if (!ret && gtdt->non_secure_el2_interrupt) + ret = mk_dtb_append_ppi(fdt, node, gtdt->non_secure_el2_interrupt, + gtdt->non_secure_el2_flags); + if (!ret && (gtdt->non_secure_el1_flags & ACPI_GTDT_ALWAYS_ON)) + ret = fdt_setprop_empty(fdt, node, "always-on"); + + acpi_put_table(table); + return ret; +} + +struct mk_gic_frames { + u64 reg[2 * (1 + MK_GICR_MAX_REGIONS)]; /* distributor first */ + int nr_gicr; + int nr_gicc; + u32 gicr_size; + bool have_gicd; +}; + +static int mk_gic_set_frame_size(struct mk_gic_frames *f) +{ + void __iomem *dist; + u32 arch; + + dist = ioremap(f->reg[0], SZ_64K); + if (!dist) + return -FDT_ERR_NOSPACE; + + arch = readl_relaxed(dist + GICD_PIDR2) & GIC_PIDR2_ARCH_MASK; + iounmap(dist); + if (arch != GIC_PIDR2_ARCH_GICv3 && arch != GIC_PIDR2_ARCH_GICv4) + return -FDT_ERR_BADVALUE; + + f->gicr_size = arch == GIC_PIDR2_ARCH_GICv4 ? 4 * SZ_64K : 2 * SZ_64K; + return 0; +} + +/* + * Redistributors are described either as GICR regions or, without any, + * by each GICC's own frame address; the first kind wins, as in the + * GIC driver's own probing. + */ +static void mk_gic_collect(struct acpi_table_madt *madt, + struct mk_gic_frames *f, u8 type) +{ + unsigned long end = (unsigned long)madt + madt->header.length; + struct acpi_subtable_header *sub; + + for (sub = (void *)(madt + 1); + (unsigned long)sub + sizeof(*sub) <= end && sub->length; + sub = (void *)sub + sub->length) { + u64 *gicr = &f->reg[2 * (1 + f->nr_gicr)]; + + if (sub->type != type) + continue; + + if (type == ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR) { + struct acpi_madt_generic_distributor *gicd = (void *)sub; + + f->reg[0] = gicd->base_address; + f->reg[1] = SZ_64K; + f->have_gicd = gicd->version == ACPI_MADT_GIC_VERSION_V3 || + gicd->version == ACPI_MADT_GIC_VERSION_V4 || + gicd->version == ACPI_MADT_GIC_VERSION_NONE; + return; + } + + if (f->nr_gicr == MK_GICR_MAX_REGIONS) + return; + + if (type == ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR) { + struct acpi_madt_generic_redistributor *r = (void *)sub; + + gicr[0] = r->base_address; + gicr[1] = r->length; + f->nr_gicr++; + } else { + struct acpi_madt_generic_interrupt *gicc = (void *)sub; + + f->nr_gicc++; + if (!gicc->gicr_base_address) + continue; + gicr[0] = gicc->gicr_base_address; + gicr[1] = f->gicr_size; + f->nr_gicr++; + } + } +} + +static int mk_dtb_add_gic(void *fdt) +{ + struct acpi_table_header *table; + struct mk_gic_frames *f; + int node, ret; + + if (ACPI_FAILURE(acpi_get_table(ACPI_SIG_MADT, 0, &table))) { + pr_err("No MADT to describe the GIC from\n"); + return -FDT_ERR_NOTFOUND; + } + + f = kzalloc(sizeof(*f), GFP_KERNEL); + if (!f) { + acpi_put_table(table); + return -FDT_ERR_NOSPACE; + } + + mk_gic_collect((void *)table, f, ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR); + mk_gic_collect((void *)table, f, ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR); + if (f->have_gicd && !f->nr_gicr) { + ret = mk_gic_set_frame_size(f); + if (!ret) + mk_gic_collect((void *)table, f, ACPI_MADT_TYPE_GENERIC_INTERRUPT); + } else { + ret = 0; + } + acpi_put_table(table); + if (ret) + goto out; + + if (!f->have_gicd || !f->nr_gicr) { + pr_err("The MADT describes no usable GICv3\n"); + ret = -FDT_ERR_NOTFOUND; + goto out; + } + + node = fdt_add_subnode(fdt, 0, "interrupt-controller"); + ret = node < 0 ? node : 0; + if (!ret) + ret = fdt_setprop_string(fdt, node, "compatible", "arm,gic-v3"); + if (!ret) + ret = fdt_setprop_empty(fdt, node, "interrupt-controller"); + if (!ret) + ret = fdt_setprop_u32(fdt, node, "#interrupt-cells", 3); + if (!ret) + ret = fdt_setprop_u32(fdt, node, "#redistributor-regions", + f->nr_gicr); + if (!ret) + ret = fdt_setprop_u32(fdt, node, "phandle", MK_GIC_PHANDLE); + if (!ret) + ret = mk_dtb_set_reg(fdt, node, f->reg, 1 + f->nr_gicr); + if (!ret) + ret = fdt_setprop_u32(fdt, 0, "interrupt-parent", MK_GIC_PHANDLE); +out: + kfree(f); + return ret; +} + +int mk_dtb_add_platform_acpi(void *fdt) +{ + int ret; + + ret = mk_dtb_add_psci(fdt); + if (!ret) + ret = mk_dtb_add_timer(fdt); + if (!ret) + ret = mk_dtb_add_gic(fdt); + return ret; +} diff --git a/arch/arm64/multikernel/dtb_of.c b/arch/arm64/multikernel/dtb_of.c new file mode 100644 index 00000000000000..6da52cf463267c --- /dev/null +++ b/arch/arm64/multikernel/dtb_of.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Platform nodes of an instance tree, on a host that booted from a + * device tree: copies of the host's own nodes. + */ + +#define pr_fmt(fmt) "mk_dtb: " fmt + +#include +#include +#include + +#include "internal.h" + +static const struct of_device_id mk_psci_ids[] = { + { .compatible = "arm,psci" }, + { .compatible = "arm,psci-0.2" }, + { .compatible = "arm,psci-1.0" }, + {} +}; + +static const struct of_device_id mk_timer_ids[] = { + { .compatible = "arm,armv8-timer" }, + { .compatible = "arm,armv7-timer" }, + {} +}; + +static bool mk_prop_is_addressing(const char *name) +{ + return !strcmp(name, "reg") || !strcmp(name, "ranges") || + !strcmp(name, "#address-cells") || !strcmp(name, "#size-cells"); +} + +/* + * Copy @np to a new node under the root and return its offset. The + * phandle comes along, so interrupt-parent references between copied + * nodes keep working. Addressing is only valid below @np's parent bus, + * so reg is written again, translated to CPU addresses. + */ +static int mk_dtb_copy_of_node(void *fdt, struct device_node *np) +{ + u64 reg[2 * MK_DTB_MAX_REG]; + struct property *pp; + struct resource res; + int node, nr = 0, ret; + + node = fdt_add_subnode(fdt, 0, np->full_name); + if (node < 0) + return node; + + for_each_property_of_node(np, pp) { + if (!strcmp(pp->name, "name") || mk_prop_is_addressing(pp->name)) + continue; + ret = fdt_setprop(fdt, node, pp->name, pp->value, pp->length); + if (ret) + return ret; + } + + while (nr < MK_DTB_MAX_REG && !of_address_to_resource(np, nr, &res)) { + reg[2 * nr] = res.start; + reg[2 * nr + 1] = resource_size(&res); + nr++; + } + if (nr) { + ret = mk_dtb_set_reg(fdt, node, reg, nr); + if (ret) + return ret; + } + return node; +} + +static int mk_dtb_copy_matching(void *fdt, const struct of_device_id *ids, + const char *what) +{ + struct device_node *np = of_find_matching_node(NULL, ids); + int node; + + if (!np) { + pr_err("The host tree has no %s node\n", what); + return -FDT_ERR_NOTFOUND; + } + node = mk_dtb_copy_of_node(fdt, np); + of_node_put(np); + return node < 0 ? node : 0; +} + +/* + * The GIC's children are left behind: an ITS has one command queue, + * which is the host's. + */ +static int mk_dtb_copy_gic(void *fdt) +{ + struct device_node *np; + int node; + + np = of_find_compatible_node(NULL, NULL, "arm,gic-v3"); + if (!np) { + pr_err("The host tree has no GICv3\n"); + return -FDT_ERR_NOTFOUND; + } + + node = mk_dtb_copy_of_node(fdt, np); + of_node_put(np); + if (node < 0) + return node; + + return fdt_setprop_u32(fdt, 0, "interrupt-parent", + fdt_get_phandle(fdt, node)); +} + +int mk_dtb_add_platform_of(void *fdt) +{ + int ret; + + ret = mk_dtb_copy_matching(fdt, mk_psci_ids, "PSCI"); + if (!ret) + ret = mk_dtb_copy_matching(fdt, mk_timer_ids, "arch timer"); + if (!ret) + ret = mk_dtb_copy_gic(fdt); + return ret; +} diff --git a/arch/arm64/multikernel/dtb_pci.c b/arch/arm64/multikernel/dtb_pci.c new file mode 100644 index 00000000000000..09ea86affcfa05 --- /dev/null +++ b/arch/arm64/multikernel/dtb_pci.c @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * PCI in an instance tree. + * + * The generic code describes the root buses above the instance's PCI + * devices, ECAM window included, which makes each of them an ordinary + * ECAM host bridge; what is left is to name the driver for it. INTx is + * not offered: the lines are SPIs that the host shares between devices. + * MSIs go through the host's ITSs, which the instance reaches through a + * proxy node. The node has no address: everything, the doorbell of the + * ITS behind a device included, comes over the message ring + * (irq-gic-v3-its-mk.c). + */ + +#define pr_fmt(fmt) "mk_dtb: " fmt + +#include +#include +#include +#include +#include + +#include "internal.h" + +#define MK_ITS_PROXY "multikernel,gic-v3-its" + +static int mk_dtb_add_its_proxy(void *fdt, u32 *phandle) +{ + int node, ret; + + ret = fdt_generate_phandle(fdt, phandle); + if (ret) + return ret; + + node = fdt_add_subnode(fdt, 0, "msi-controller"); + if (node < 0) + return node; + + ret = fdt_setprop_string(fdt, node, "compatible", MK_ITS_PROXY); + if (!ret) + ret = fdt_setprop_empty(fdt, node, "msi-controller"); + if (!ret) + ret = fdt_setprop_u32(fdt, node, "#msi-cells", 1); + if (!ret) + ret = fdt_setprop_u32(fdt, node, "phandle", *phandle); + return ret; +} + +/* Every requester ID maps to itself: the host knows the real DeviceID */ +static int mk_dtb_set_msi_map(void *fdt, int node, u32 phandle) +{ + const u32 map[] = { 0, phandle, 0, SZ_64K }; + int i, ret; + + fdt_delprop(fdt, node, "msi-map"); + for (i = 0; i < ARRAY_SIZE(map); i++) { + ret = fdt_appendprop_u32(fdt, node, "msi-map", map[i]); + if (ret) + return ret; + } + return 0; +} + +/* + * What a root bus node has to say about the devices behind it, asked of a + * device on the host's side of the same bus. Devices inherit DMA coherency + * from their host bridge node, and a tree that is silent about it makes + * them all non-coherent. MSIs need an ITS behind the bus; which one does + * not matter here, the host picks it per device when an event is mapped. + */ +struct mk_root_bus_traits { + bool coherent; + bool msi; +}; + +static int mk_dtb_first_dev_traits(struct pci_dev *pdev, void *data) +{ + struct mk_root_bus_traits *traits = data; + + traits->coherent = dev_is_dma_coherent(&pdev->dev); + traits->msi = IS_ENABLED(CONFIG_MULTIKERNEL_ITS_PROXY) && + its_foreign_doorbell(dev_get_msi_domain(&pdev->dev)); + return 1; +} + +static struct mk_root_bus_traits mk_dtb_root_bus_traits(void *fdt, int node) +{ + const fdt32_t *domain = fdt_getprop(fdt, node, "linux,pci-domain", NULL); + const fdt32_t *range = fdt_getprop(fdt, node, "bus-range", NULL); + struct mk_root_bus_traits traits = {}; + struct pci_bus *bus = NULL; + + if (range) + bus = pci_find_bus(domain ? fdt32_to_cpu(*domain) : 0, + fdt32_to_cpu(range[0])); + if (bus) + pci_walk_bus(bus, mk_dtb_first_dev_traits, &traits); + return traits; +} + +#define mk_dtb_for_each_root_bus(fdt, node) \ + for (node = fdt_node_offset_by_compatible(fdt, -1, MK_PCI_HOST_BRIDGE); \ + node >= 0; \ + node = fdt_node_offset_by_compatible(fdt, node, MK_PCI_HOST_BRIDGE)) + +int mk_dtb_add_pci(void *fdt) +{ + static const char compatible[] = + MK_PCI_HOST_BRIDGE "\0pci-host-ecam-generic"; + struct mk_root_bus_traits traits; + bool any_msi = false; + u32 phandle = 0; + int node, ret; + + /* Adding the proxy node moves the root buses, so look before writing */ + mk_dtb_for_each_root_bus(fdt, node) + any_msi |= mk_dtb_root_bus_traits(fdt, node).msi; + if (any_msi) { + ret = mk_dtb_add_its_proxy(fdt, &phandle); + if (ret) + return ret; + } + + mk_dtb_for_each_root_bus(fdt, node) { + if (!fdt_getprop(fdt, node, "reg", NULL)) { + pr_warn("A PCI root bus of the instance has no ECAM window\n"); + continue; + } + + traits = mk_dtb_root_bus_traits(fdt, node); + if (!traits.msi) + pr_warn("No ITS behind a PCI root bus of the instance: its devices get no interrupts\n"); + + ret = fdt_setprop(fdt, node, "compatible", compatible, + sizeof(compatible)); + if (!ret && traits.msi) + ret = mk_dtb_set_msi_map(fdt, node, phandle); + if (!ret && traits.coherent) + ret = fdt_setprop_empty(fdt, node, "dma-coherent"); + if (ret) + return ret; + } + return 0; +} diff --git a/arch/arm64/multikernel/internal.h b/arch/arm64/multikernel/internal.h new file mode 100644 index 00000000000000..cfe073b1919dcc --- /dev/null +++ b/arch/arm64/multikernel/internal.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ARM64_MULTIKERNEL_INTERNAL_H +#define _ARM64_MULTIKERNEL_INTERNAL_H + +#include +#include + +struct kimage; +struct mk_instance; + +/* GIC interrupt specifier cells, as in dt-bindings/interrupt-controller/arm-gic.h */ +#define MK_PCI_HOST_BRIDGE "multikernel,pci-host-bridge" + +#define MK_GIC_PPI 1 +#define MK_GIC_PHANDLE 1 + +int mk_build_boot_dtb(struct kimage *image, struct mk_instance *instance); + +int mk_dtb_set_reg(void *fdt, int node, const u64 *pairs, int nr); + +/* reg entries a copied node may have; a GIC has one per redistributor region */ +#define MK_DTB_MAX_REG 16 + +/* The driver and the MSI controller for the instance's PCI root buses */ +int mk_dtb_add_pci(void *fdt); + +/* + * The part of the machine every kernel drives for itself: PSCI, the arch + * timer and the interrupt controller. Taken from whichever description + * this kernel booted with. + */ +int mk_dtb_add_platform_of(void *fdt); +#ifdef CONFIG_ACPI +int mk_dtb_add_platform_acpi(void *fdt); +#else +static inline int mk_dtb_add_platform_acpi(void *fdt) +{ + return -FDT_ERR_NOTFOUND; +} +#endif + +/* Force halt through SDEI, where firmware has it */ +#ifdef CONFIG_ARM_SDE_INTERFACE +int mk_dtb_add_sdei(void *fdt); +void mk_sdei_stop_cpu(struct mk_instance *instance, u64 mpidr); +void mk_sdei_unreachable(void); +#else +static inline int mk_dtb_add_sdei(void *fdt) +{ + return 0; +} + +static inline void mk_sdei_stop_cpu(struct mk_instance *instance, u64 mpidr) +{ +} + +static inline void mk_sdei_unreachable(void) +{ +} +#endif + +#endif /* _ARM64_MULTIKERNEL_INTERNAL_H */ diff --git a/arch/arm64/multikernel/its_proxy.c b/arch/arm64/multikernel/its_proxy.c new file mode 100644 index 00000000000000..eb279932ec33bc --- /dev/null +++ b/arch/arm64/multikernel/its_proxy.c @@ -0,0 +1,421 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * MSIs for an instance's PCI devices, through the host's ITS. + * + * A GICv3 ITS has a single command queue, which belongs to the kernel + * that booted the machine. An instance therefore cannot map the MSIs of + * the devices it was given, and asks over the message ring instead: + * + * - MK_IO_MSI_MAP names a device, an event and one of the instance's + * CPUs. The host programs the ITS behind the device and answers with + * the LPI that will arrive there and the doorbell to write the event + * to. The instance asks from process context and sleeps; the host works + * in a work item, because the ITS driver sleeps too. + * - MK_IO_MSI_MOVE moves a mapped event to another CPU. It comes from an + * interrupt chip callback, with the descriptor locked and possibly from + * a dying CPU inside stop_machine(), and that CPU may be turned off the + * moment the callback returns. So the instance waits for the answer by + * polling the ring, and the host moves the event right in its doorbell + * interrupt, which takes a cache of what MAP resolved: neither the PCI + * device nor the instance can be looked up there. + * + * Both answers are MK_IO_MSI_ACK. There is no unmap: the host drops + * everything when the instance halts, and again before it is started. + * + * Both ends live here. The ITS side of the host is its_foreign_*() in the + * ITS driver, the interrupt domain of the instance is irq-gic-v3-its-mk. + */ + +#define pr_fmt(fmt) "mk_its: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MK_MSI_MAP_TIMEOUT_MS 5000 +/* As long as the ITS driver gives one of its own commands */ +#define MK_MSI_MOVE_TIMEOUT_US USEC_PER_SEC +#define MK_MSI_MOVE_POLL_US 10 + +/* + * @rid is the device as the instance sees it on its PCI bus, @event the + * MSI data it will write. + */ +struct mk_io_msi_payload { + u32 request; /* MK_IO_MSI_MAP or MK_IO_MSI_MOVE, echoed in the ACK */ + u32 domain; /* PCI domain */ + u32 rid; /* bus << 8 | devfn */ + u32 event; + u64 phys_cpu; /* Target CPU, one of the sender's */ + u64 doorbell; /* ACK of a MAP: address the device writes the event to */ + int result; /* ACK: LPI INTID of a MAP, 0 of a MOVE, or -errno */ + int sender_instance_id; +}; + +/* An answer belongs to the request it echoes */ +static bool mk_msi_same_request(const struct mk_io_msi_payload *a, + const struct mk_io_msi_payload *b) +{ + return a->request == b->request && a->domain == b->domain && + a->rid == b->rid && a->event == b->event; +} + +/* + * Host side + */ + +struct mk_msi_work { + struct work_struct work; + struct mk_io_msi_payload req; +}; + +/* What a MAP resolved about a device, for the MOVEs that follow */ +struct mk_msi_route { + struct list_head list; + struct mk_instance *instance; + u32 domain; + u32 rid; + struct irq_domain *msi_domain; + u32 dev_id; +}; + +static LIST_HEAD(mk_msi_routes); +static DEFINE_RAW_SPINLOCK(mk_msi_routes_lock); + +static struct mk_msi_route *mk_msi_find_route(int instance_id, u32 domain, u32 rid) +{ + struct mk_msi_route *route; + + list_for_each_entry(route, &mk_msi_routes, list) { + if (route->instance->id == instance_id && + route->domain == domain && route->rid == rid) + return route; + } + return NULL; +} + +static int mk_msi_remember_route(struct mk_instance *instance, + const struct mk_io_msi_payload *req, + struct irq_domain *msi_domain, u32 dev_id) +{ + struct mk_msi_route *route; + + scoped_guard(raw_spinlock_irqsave, &mk_msi_routes_lock) { + if (mk_msi_find_route(instance->id, req->domain, req->rid)) + return 0; + } + + route = kzalloc_obj(*route); + if (!route) + return -ENOMEM; + route->instance = mk_instance_get(instance); + route->domain = req->domain; + route->rid = req->rid; + route->msi_domain = msi_domain; + route->dev_id = dev_id; + + /* One work item at a time maps for a device, so nobody got in between */ + scoped_guard(raw_spinlock_irqsave, &mk_msi_routes_lock) + list_add(&route->list, &mk_msi_routes); + return 0; +} + +/* Every vector the device has: the instance's driver may use any of them */ +static u32 mk_msi_max_vectors(struct pci_dev *pdev) +{ + return max3(pci_msix_vec_count(pdev), pci_msi_vec_count(pdev), 1); +} + +/* Only a device the instance was given, only a CPU it owns */ +static int mk_msi_map_for(struct mk_instance *instance, + struct mk_io_msi_payload *req) +{ + struct irq_domain *msi_domain; + struct mk_pci_device *mine; + phys_addr_t doorbell = 0; + struct pci_dev *pdev; + bool owned = false; + int cpu, ret; + u32 dev_id; + + list_for_each_entry(mine, &instance->pci_devices, list) { + if (mine->domain == req->domain && mine->bus == req->rid >> 8 && + PCI_DEVFN(mine->slot, mine->func) == (req->rid & 0xff)) { + owned = true; + break; + } + } + if (!owned || !mk_cpu_set_contains(instance->cpus, req->phys_cpu)) + return -EPERM; + + cpu = arch_cpu_from_physical_id(req->phys_cpu); + if (cpu < 0) + return -EINVAL; + + pdev = pci_get_domain_bus_and_slot(req->domain, req->rid >> 8, + req->rid & 0xff); + if (!pdev) + return -ENODEV; + + /* The device's MSI domain names the ITS behind it, of possibly several */ + msi_domain = dev_get_msi_domain(&pdev->dev); + dev_id = pci_msi_domain_get_msi_rid(msi_domain, pdev); + ret = its_foreign_map(instance, msi_domain, dev_id, req->event, + mk_msi_max_vectors(pdev), cpu, &doorbell); + pci_dev_put(pdev); + if (ret < 0) + return ret; + + req->doorbell = doorbell; + return mk_msi_remember_route(instance, req, msi_domain, dev_id) ?: ret; +} + +static void mk_msi_map_work(struct work_struct *work) +{ + struct mk_msi_work *w = container_of(work, struct mk_msi_work, work); + struct mk_io_msi_payload *req = &w->req; + struct mk_instance *instance; + int to = req->sender_instance_id; + + instance = mk_instance_find(to); + req->result = instance ? mk_msi_map_for(instance, req) : -ENOENT; + if (req->result < 0) + pr_err("Instance %d: no MSI for %04x:%02x:%02x.%x event %u: %d\n", + to, req->domain, req->rid >> 8, PCI_SLOT(req->rid), + PCI_FUNC(req->rid), req->event, req->result); + + req->sender_instance_id = mk_self->id; + if (instance) { + mk_send_message_to(instance, MK_MSG_IO, MK_IO_MSI_ACK, req, + sizeof(*req)); + mk_instance_put(instance); + } + kfree(w); +} + +/* + * In the doorbell interrupt. The route keeps the instance alive, and the + * lock keeps the route: mk_arch_msi_release() takes it to unlink. Without + * a route there is nobody to answer: the instance never mapped the device, + * or has halted since, and its request times out. + */ +static void mk_msi_move(const struct mk_io_msi_payload *req) +{ + struct mk_io_msi_payload ack = *req; + struct mk_msi_route *route; + int cpu; + + guard(raw_spinlock_irqsave)(&mk_msi_routes_lock); + + route = mk_msi_find_route(req->sender_instance_id, req->domain, req->rid); + if (!route) + return; + + cpu = arch_cpu_from_physical_id(req->phys_cpu); + if (cpu < 0 || !mk_cpu_set_contains(route->instance->cpus, req->phys_cpu)) + ack.result = -EPERM; + else + ack.result = its_foreign_move(route->instance, route->msi_domain, + route->dev_id, req->event, cpu); + + ack.sender_instance_id = mk_self->id; + mk_send_message_to(route->instance, MK_MSG_IO, MK_IO_MSI_ACK, &ack, + sizeof(ack)); +} + +void mk_arch_msi_release(struct mk_instance *instance) +{ + struct mk_msi_route *route, *tmp; + LIST_HEAD(gone); + + scoped_guard(raw_spinlock_irqsave, &mk_msi_routes_lock) { + list_for_each_entry_safe(route, tmp, &mk_msi_routes, list) { + if (route->instance == instance) + list_move(&route->list, &gone); + } + } + + list_for_each_entry_safe(route, tmp, &gone, list) { + mk_instance_put(route->instance); + kfree(route); + } + + its_foreign_release(instance); +} + +phys_addr_t mk_arch_msi_doorbell(struct pci_dev *pdev) +{ + return its_foreign_doorbell(dev_get_msi_domain(&pdev->dev)); +} + +/* + * Instance side + */ + +/* On the requester's stack until the answer is in, or given up on */ +struct mk_msi_request { + struct list_head list; + struct completion done; + struct mk_io_msi_payload msg; /* the request, then its answer */ +}; + +static LIST_HEAD(mk_msi_requests); +static DEFINE_RAW_SPINLOCK(mk_msi_requests_lock); + +static void mk_msi_answer(const struct mk_io_msi_payload *answer) +{ + struct mk_msi_request *r; + + guard(raw_spinlock_irqsave)(&mk_msi_requests_lock); + + list_for_each_entry(r, &mk_msi_requests, list) { + if (mk_msi_same_request(&r->msg, answer)) { + r->msg = *answer; + complete(&r->done); + return; + } + } +} + +static bool mk_msi_wait_sleeping(struct mk_msi_request *r) +{ + return wait_for_completion_timeout(&r->done, + msecs_to_jiffies(MK_MSI_MAP_TIMEOUT_MS)); +} + +/* + * Nobody may be there to take the doorbell: this CPU has interrupts off, + * and under stop_machine() so has every other. Drain the ring from here. + */ +static bool mk_msi_wait_polling(struct mk_msi_request *r) +{ + unsigned int waited; + + for (waited = 0; waited < MK_MSI_MOVE_TIMEOUT_US; + waited += MK_MSI_MOVE_POLL_US) { + mk_ipi_poll(); + if (try_wait_for_completion(&r->done)) + return true; + udelay(MK_MSI_MOVE_POLL_US); + } + return false; +} + +static int mk_msi_request(struct mk_io_msi_payload *req, + bool (*wait)(struct mk_msi_request *r)) +{ + struct mk_msi_request r; + int ret; + + if (!host_instance) + return -ENODEV; + req->sender_instance_id = mk_self->id; + r.msg = *req; + init_completion(&r.done); + + scoped_guard(raw_spinlock_irqsave, &mk_msi_requests_lock) + list_add(&r.list, &mk_msi_requests); + + ret = mk_send_message_to(host_instance, MK_MSG_IO, req->request, req, + sizeof(*req)); + if (!ret && !wait(&r)) + ret = -ETIMEDOUT; + + /* Off the list before the stack frame goes: a late answer finds nothing */ + scoped_guard(raw_spinlock_irqsave, &mk_msi_requests_lock) + list_del(&r.list); + + if (ret) + return ret; + *req = r.msg; + return req->result; +} + +/** + * mk_msi_proxy_map - Ask the host to route a device's MSI to a CPU + * @domain: PCI domain of the device + * @rid: bus << 8 | devfn + * @event: MSI data the device will write + * @phys_cpu: target, one of this kernel's CPUs + * @doorbell: set to the address to write @event to + * + * Sleeps. Returns the interrupt number or a negative error code. + */ +int mk_msi_proxy_map(u32 domain, u32 rid, u32 event, u64 phys_cpu, + phys_addr_t *doorbell) +{ + struct mk_io_msi_payload req = { + .request = MK_IO_MSI_MAP, + .domain = domain, + .rid = rid, + .event = event, + .phys_cpu = phys_cpu, + }; + int ret = mk_msi_request(&req, mk_msi_wait_sleeping); + + if (ret >= 0) + *doorbell = req.doorbell; + return ret; +} + +/** + * mk_msi_proxy_move - Ask the host to move a routed MSI to another CPU + * @domain: PCI domain of the device + * @rid: bus << 8 | devfn + * @event: event that mk_msi_proxy_map() has routed + * @phys_cpu: new target, one of this kernel's CPUs + * + * Returns once the host has moved it, so the CPU it came from may go + * away. Any context. Returns 0 or a negative error code. + */ +int mk_msi_proxy_move(u32 domain, u32 rid, u32 event, u64 phys_cpu) +{ + struct mk_io_msi_payload req = { + .request = MK_IO_MSI_MOVE, + .domain = domain, + .rid = rid, + .event = event, + .phys_cpu = phys_cpu, + }; + + return mk_msi_request(&req, mk_msi_wait_polling); +} + +static void mk_msi_msg_handler(u32 msg_type, u32 subtype, void *payload, + u32 payload_len, void *ctx) +{ + struct mk_io_msi_payload *p = payload; + struct mk_msi_work *w; + + if (payload_len < sizeof(*p)) + return; + + switch (subtype) { + case MK_IO_MSI_MAP: + w = kzalloc(sizeof(*w), GFP_ATOMIC); + if (!w) + return; + w->req = *p; + INIT_WORK(&w->work, mk_msi_map_work); + schedule_work(&w->work); + break; + case MK_IO_MSI_MOVE: + mk_msi_move(p); + break; + case MK_IO_MSI_ACK: + mk_msi_answer(p); + break; + } +} + +static int __init mk_msi_proxy_init(void) +{ + return mk_register_msg_handler(MK_MSG_IO, mk_msi_msg_handler, NULL); +} +fs_initcall(mk_msi_proxy_init); diff --git a/arch/arm64/multikernel/sdei.c b/arch/arm64/multikernel/sdei.c new file mode 100644 index 00000000000000..3a09a2b5ea186e --- /dev/null +++ b/arch/arm64/multikernel/sdei.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Force halt through SDEI. + * + * The stop SGI reaches a CPU that spins with interrupts masked only when + * its kernel runs with pseudo-NMIs, and never one that has masked every + * exception. SDEI's software-signalled event 0 is taken at EL3 and turned + * into a call of the handler its kernel registered, whatever that CPU has + * masked. So every kernel registers it, and a force halt signals it to the + * CPUs of an instance before it sends them the SGI. + * + * TF-A acknowledges event 0 on a CPU that has SDEI masked and never ends + * it, which leaves that CPU unable to take any non-secure interrupt. A CPU + * therefore publishes in fw_stop[] of its shared area whether it can take + * the event, and clears that before anything masks SDEI: going off, and + * around a power-down idle state. + */ + +#define pr_fmt(fmt) "mk_sdei: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +/* The software-signalled event, the only one SDEI_EVENT_SIGNAL raises */ +#define MK_SDEI_STOP_EVENT 0 + +static DEFINE_PER_CPU(u8 *, mk_fw_stop); + +#ifdef CONFIG_ACPI +static bool mk_acpi_has_sdei(void) +{ + struct acpi_table_header *table; + + if (acpi_disabled || ACPI_FAILURE(acpi_get_table(ACPI_SIG_SDEI, 0, &table))) + return false; + acpi_put_table(table); + return true; +} +#else +static bool mk_acpi_has_sdei(void) +{ + return false; +} +#endif + +/* The conduit to firmware's SDEI, as the SDEI driver finds it; NULL without */ +static const char *mk_sdei_method(void) +{ + struct device_node *np; + const char *method = NULL; + + np = of_find_compatible_node(NULL, NULL, "arm,sdei-1.0"); + if (np) { + of_property_read_string(np, "method", &method); + of_node_put(np); + } else if (mk_acpi_has_sdei()) { + method = acpi_psci_use_hvc() ? "hvc" : "smc"; + } + return method; +} + +int mk_dtb_add_sdei(void *fdt) +{ + const char *method = mk_sdei_method(); + int node, ret; + + if (!method) + return 0; + + node = fdt_subnode_offset(fdt, 0, "firmware"); + if (node == -FDT_ERR_NOTFOUND) + node = fdt_add_subnode(fdt, 0, "firmware"); + if (node >= 0) + node = fdt_add_subnode(fdt, node, "sdei"); + if (node < 0) + return node; + + ret = fdt_setprop_string(fdt, node, "compatible", "arm,sdei-1.0"); + if (!ret) + ret = fdt_setprop_string(fdt, node, "method", method); + return ret; +} + +static void mk_sdei_set_reachable(bool reachable) +{ + u8 *fw_stop = this_cpu_read(mk_fw_stop); + + if (fw_stop) + WRITE_ONCE(*fw_stop, reachable); +} + +void mk_sdei_unreachable(void) +{ + mk_sdei_set_reachable(false); +} + +void mk_sdei_stop_cpu(struct mk_instance *instance, u64 mpidr) +{ + struct mk_shared_data *sd = instance->ipi_data; + int rank = mk_cpu_rank(mpidr); + int ret; + + if (!sd || rank >= MK_PARKED_MAX || !READ_ONCE(sd->fw_stop[rank])) + return; + + ret = sdei_event_signal(MK_SDEI_STOP_EVENT, mpidr); + if (ret) + pr_err_ratelimited("No SDEI stop to CPU 0x%llx: %d\n", mpidr, ret); +} + +static int mk_sdei_stop(u32 event, struct pt_regs *regs, void *arg) +{ + if (!mk_has_pending_shutdown()) { + pr_warn_ratelimited("CPU%u: ignoring an SDEI stop from another kernel\n", + smp_processor_id()); + return 0; + } + + /* + * Complete the event before the CPU goes off. TF-A keeps the + * priority of a running handler active, and panics when the CPU + * takes the next event on top of it after its next CPU_ON. + */ + __sdei_handler_abort(); + __this_cpu_write(sdei_active_normal_event, NULL); + + mk_foreign_cpu_stop(); + return 0; +} + +/* + * The SDEI driver's own states and notifier mask and unmask the CPU. Its + * hotplug state comes first, so this one is set up after it unmasks and + * torn down before it masks; the PM notifiers sit on either side of its. + */ +static int mk_sdei_cpu_up(unsigned int cpu) +{ + struct mk_shared_data *sd = mk_self ? mk_self->ipi_data : NULL; + int rank = mk_cpu_rank(arch_cpu_physical_id(cpu)); + + if (sd && rank < MK_PARKED_MAX) + this_cpu_write(mk_fw_stop, &sd->fw_stop[rank]); + mk_sdei_set_reachable(true); + return 0; +} + +static int mk_sdei_cpu_down(unsigned int cpu) +{ + mk_sdei_set_reachable(false); + return 0; +} + +static int mk_sdei_pm_enter(struct notifier_block *nb, unsigned long action, + void *data) +{ + if (action == CPU_PM_ENTER) + mk_sdei_set_reachable(false); + else if (action == CPU_PM_ENTER_FAILED) + mk_sdei_set_reachable(true); + return NOTIFY_OK; +} + +static int mk_sdei_pm_exit(struct notifier_block *nb, unsigned long action, + void *data) +{ + if (action == CPU_PM_EXIT) + mk_sdei_set_reachable(true); + return NOTIFY_OK; +} + +static struct notifier_block mk_sdei_pm_enter_nb = { + .notifier_call = mk_sdei_pm_enter, + .priority = 1, +}; + +static struct notifier_block mk_sdei_pm_exit_nb = { + .notifier_call = mk_sdei_pm_exit, + .priority = -1, +}; + +static int __init mk_sdei_init(void) +{ + int ret; + + if (!mk_sdei_method()) + return 0; + + ret = sdei_event_register(MK_SDEI_STOP_EVENT, mk_sdei_stop, NULL); + if (ret) + goto out; + ret = sdei_event_enable(MK_SDEI_STOP_EVENT); + if (ret) + goto unregister; + + cpu_pm_register_notifier(&mk_sdei_pm_enter_nb); + cpu_pm_register_notifier(&mk_sdei_pm_exit_nb); + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/multikernel/sdei:online", + mk_sdei_cpu_up, mk_sdei_cpu_down); + if (ret >= 0) { + pr_info("Force halt stops this kernel's CPUs through SDEI\n"); + return 0; + } + + cpu_pm_unregister_notifier(&mk_sdei_pm_exit_nb); + cpu_pm_unregister_notifier(&mk_sdei_pm_enter_nb); + sdei_event_disable(MK_SDEI_STOP_EVENT); +unregister: + sdei_event_unregister(MK_SDEI_STOP_EVENT); +out: + pr_warn("No force halt through SDEI: %d\n", ret); + return 0; +} +late_initcall(mk_sdei_init); diff --git a/arch/arm64/multikernel/spawn.c b/arch/arm64/multikernel/spawn.c new file mode 100644 index 00000000000000..a3c8c0f63f09d1 --- /dev/null +++ b/arch/arm64/multikernel/spawn.c @@ -0,0 +1,364 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * arm64 multikernel architecture interface. + * + * Firmware owns CPU state on arm64. A pool CPU is a CPU in the PSCI OFF + * state: it executes nothing, so there is no park loop to keep in memory + * and no mailbox to watch. CPU_ON is the boot trampoline, delivering the + * CPU to an Image entry point in exactly the state head.S requires, and + * AFFINITY_INFO tells whether a CPU has come back. + */ + +#define pr_fmt(fmt) "mk_spawn: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +/* A halting kernel acknowledges first and turns its CPUs off afterwards */ +#define MK_CPU_OFF_TIMEOUT_US USEC_PER_SEC + +/* + * Ring the doorbell of a CPU that another kernel runs on. There is no + * foreign doorbell to filter on the receiving side: all it does is + * drain this kernel's own ring, which such a sender cannot have filled. + */ +void mk_arch_send_ipi(mk_phys_cpu_t phys_cpu) +{ + int ret = gic_v3_send_sgi_to_mpidr(phys_cpu, IPI_MULTIKERNEL); + + if (ret) + pr_err_ratelimited("No doorbell SGI to CPU 0x%llx: %d\n", + phys_cpu, ret); +} + +void __init mk_arch_register_cpu(mk_phys_cpu_t phys_id) +{ +} + +void __init mk_spawn_accept_boot_tree(phys_addr_t dt_phys) +{ + if (!initial_boot_params || + !of_flat_dt_is_compatible(of_get_flat_dt_root(), MK_FDT_COMPATIBLE)) + return; + + mk_manifest_populate(dt_phys, fdt_totalsize(initial_boot_params)); +} + +/* + * Return the calling CPU to firmware. The boot CPU is no exception: the + * host's CPUs are on, so it is never the last one firmware knows about. + */ +void __noreturn mk_enter_pool_state(void *info) +{ + unsigned int cpu = smp_processor_id(); + const struct cpu_operations *ops = get_cpu_ops(cpu); + + local_daif_mask(); + mk_sdei_unreachable(); + sdei_mask_local_cpu(); + + if (ops && ops->cpu_die) + ops->cpu_die(cpu); + + pr_crit("CPU%u could not be turned off, the host cannot reuse it\n", cpu); + cpu_park_loop(); +} + +void mk_spawn_machine_halt(void) +{ + if (mk_spawned()) + mk_halt_to_pool(); +} + +void mk_spawn_stop_this_cpu(void) +{ + if (mk_spawned()) + mk_enter_pool_state(NULL); +} + +/* + * A panicked spawn kernel spinning in the panic loop holds its CPUs + * hostage. "Reboot on panic" is mk_spawn_machine_halt() here, which + * notifies the host and turns the CPUs off, so make it the default. + */ +static int __init mk_spawn_reboot_init(void) +{ + if (mk_spawned() && !panic_timeout) + panic_timeout = -1; + return 0; +} +core_initcall(mk_spawn_reboot_init); + +/* + * PSCI cannot turn another CPU off, so a kernel that no longer answers + * has to be made to turn its CPUs off itself. SDEI reaches a CPU whatever + * it has masked, but only one that said it can take the event. The stop + * IPI its own smp_send_stop() would use goes to every CPU regardless: as + * IPI_CPU_STOP_NMI a kernel running with pseudo-NMIs takes it even with + * interrupts masked, and to any other kernel it is an ordinary SGI. + */ +void mk_force_stop_cpu(struct mk_instance *instance, mk_phys_cpu_t phys_cpu) +{ + int ret; + + mk_sdei_stop_cpu(instance, phys_cpu); + + ret = gic_v3_send_sgi_to_mpidr(phys_cpu, IPI_CPU_STOP_NMI); + if (ret) + pr_err("No stop SGI to CPU 0x%llx: %d\n", phys_cpu, ret); +} + +/* + * The receiving end. A force halt is announced in the shared ring area + * before the SGIs go out, and only a kernel this one has a relationship + * with can write there; a stop from anyone else is ignored, the way x86 + * drops a stop NMI it did not ask for. Obeying means leaving for good: + * the CPU is turned off, in a host that is being fenced as well. + */ +void mk_foreign_cpu_stop(void) +{ + if (!mk_has_pending_shutdown()) { + pr_warn_ratelimited("CPU%u: ignoring a stop IPI from another kernel\n", + smp_processor_id()); + return; + } + + set_cpu_online(smp_processor_id(), false); + mk_enter_pool_state(NULL); +} + +static struct mk_cpu_set *mk_started_set(struct mk_instance *instance) +{ + if (!instance->cpus_on_slot) + instance->cpus_on_slot = mk_cpu_set_alloc(); + + return instance->cpus_on_slot; +} + +static int mk_wait_cpu_off(mk_phys_cpu_t mpidr) +{ + int state; + + if (!psci_ops.affinity_info) + return -EOPNOTSUPP; + + return read_poll_timeout(psci_ops.affinity_info, state, + state == PSCI_0_2_AFFINITY_LEVEL_OFF, + 1000, MK_CPU_OFF_TIMEOUT_US, false, mpidr, 0); +} + +/* + * The spawn reads these with the MMU and caches off, so what the host + * wrote through its cacheable mapping has to be in memory by then. + */ +static void mk_clean_to_poc(phys_addr_t start, size_t size) +{ + unsigned long va = (unsigned long)phys_to_virt(start); + + dcache_clean_inval_poc(va, va + size); +} + +/* + * Everything the new kernel reads before its MMU is on is a segment: the + * Image and the device tree. The manifest and the message ring are only + * touched with the caches on, where both kernels see the same memory. + */ +static void mk_clean_image_to_poc(struct kimage *image) +{ + unsigned long i; + + for (i = 0; i < image->nr_segments; i++) + mk_clean_to_poc(image->segment[i].mem, image->segment[i].memsz); +} + +/** + * mk_arch_spawn_instance - Start an instance's boot CPU on its image + * @image: Loaded kimage for the instance + * @instance: Instance being spawned + * @cpu: Logical CPU, off in the pool, to boot the instance on + * + * Once the boot CPU runs, the instance's kernel may turn on any other CPU + * it was given without telling the host, so all of them count as started + * from here on and have to be seen off again before the image is reused. + */ +int mk_arch_spawn_instance(struct kimage *image, struct mk_instance *instance, + int cpu) +{ + struct mk_cpu_set *started = mk_started_set(instance); + mk_phys_cpu_t mpidr = arch_cpu_physical_id(cpu); + mk_phys_cpu_t phys_cpu; + unsigned int i; + int ret; + + if (!mk_pool) + return -ENODEV; + + if (!image->arch.dtb_mem) { + pr_err("Instance %d has no boot device tree\n", instance->id); + return -EINVAL; + } + + /* Make the adds below infallible before anything runs */ + if (!started) + return -ENOMEM; + ret = mk_cpu_set_reserve(started, mk_cpu_set_count(instance->cpus)); + if (ret) + return ret; + + ret = mk_build_boot_dtb(image, instance); + if (ret) { + pr_err("No boot device tree for instance %d: %d\n", + instance->id, ret); + return ret; + } + + mk_clean_image_to_poc(image); + + guard(mutex)(&mk_pool->park_lock); + + ret = psci_cpu_on_context(mpidr, image->start, image->arch.dtb_mem); + if (ret) { + pr_err("CPU_ON of CPU 0x%llx for instance %d failed: %d\n", + mpidr, instance->id, ret); + return ret; + } + + mk_cpu_set_for_each(i, phys_cpu, instance->cpus) + mk_cpu_set_add(started, phys_cpu); + + return 0; +} + +/** + * mk_arch_release_instance - Let go of an instance's CPUs before teardown + * @instance: Instance being torn down + * + * Fails if a CPU is still on: it may be executing the instance's image, + * so the caller must keep the instance's memory alive. + */ +int mk_arch_release_instance(struct mk_instance *instance) +{ + int ret; + + ret = mk_repark_instance_to_host(instance); + if (ret) { + char buf[256]; + + mk_cpu_set_format(buf, sizeof(buf), instance->cpus_on_slot); + pr_err("Instance %d (%s): CPUs %s are not off, keeping its memory\n", + instance->id, instance->name, buf); + return ret; + } + + mk_cpu_set_free(instance->cpus_on_slot); + instance->cpus_on_slot = NULL; + return 0; +} + +/** + * mk_arch_confirm_parked - Verify a CPU has left the instance's kernel + * @instance: Instance the CPU belongs to + * @phys_cpu: MPIDR of the CPU + * + * Returns 0 once firmware reports the CPU off. + */ +int mk_arch_confirm_parked(struct mk_instance *instance, mk_phys_cpu_t phys_cpu) +{ + return mk_wait_cpu_off(phys_cpu); +} + +/** + * mk_repark_instance_to_host - Return an instance's CPUs to the pool + * @instance: Instance being torn down + * + * An off CPU belongs to whoever turns it on next, so there is nothing to + * move. What is left to do is to see every started CPU off; one that is + * not stays tracked, and the caller must not free the instance's memory. + */ +int mk_repark_instance_to_host(struct mk_instance *instance) +{ + struct mk_cpu_set *started = instance->cpus_on_slot; + unsigned int i; + int failed = 0; + + if (mk_cpu_set_empty(started)) + return 0; + + guard(mutex)(&mk_pool->park_lock); + + /* Back to front, so deleting never shifts entries yet to be visited */ + for (i = mk_cpu_set_count(started); i-- > 0; ) { + mk_phys_cpu_t phys_cpu = started->ids[i]; + + if (mk_wait_cpu_off(phys_cpu)) + failed++; + else + mk_cpu_set_del(started, phys_cpu); + } + + return failed ? -ETIMEDOUT : 0; +} + +/** + * mk_repark_cpu_to_instance - Hand one pool CPU to a live instance + * @instance: Running instance the CPU is being hot-added to + * @phys_cpu: MPIDR of the CPU + * + * The instance's kernel turns the CPU on by itself; the host only starts + * counting it among the CPUs that kernel may be running on. + */ +int mk_repark_cpu_to_instance(struct mk_instance *instance, mk_phys_cpu_t phys_cpu) +{ + struct mk_cpu_set *started = mk_started_set(instance); + + if (!started) + return -ENOMEM; + + guard(mutex)(&mk_pool->park_lock); + + if (mk_cpu_set_contains(started, phys_cpu)) + return 0; + + return mk_cpu_set_add(started, phys_cpu); +} + +/** + * mk_repark_cpu_to_host - Take one CPU back from an instance + * @instance: Instance the CPU was removed from + * @phys_cpu: MPIDR of the CPU + * + * The instance's kernel reports the CPU offline before the CPU has + * reached firmware. Turning it on elsewhere only works once it is off, + * so wait for that; until then it stays accounted to the instance. + */ +int mk_repark_cpu_to_host(struct mk_instance *instance, mk_phys_cpu_t phys_cpu) +{ + int ret; + + guard(mutex)(&mk_pool->park_lock); + + if (!mk_cpu_set_contains(instance->cpus_on_slot, phys_cpu)) + return 0; + + ret = mk_wait_cpu_off(phys_cpu); + if (!ret) + mk_cpu_set_del(instance->cpus_on_slot, phys_cpu); + return ret; +} diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index a4bda8e87536c1..fba3b174227d21 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -446,6 +446,7 @@ device_initcall(mk_register_force_stop); /** * mk_force_stop_cpu - Send NMI to a specific CPU to force it to stop + * @instance: Instance the CPU belongs to * @phys_cpu: Physical CPU ID to stop * * The target belongs to a different kernel instance, so the NMI goes to @@ -454,7 +455,7 @@ device_initcall(mk_register_force_stop); * through it silently dropped the NMI for CPUs it missed. The ICR must * drain first, or back-to-back sends lose all but the first on xAPIC. */ -void mk_force_stop_cpu(u64 phys_cpu) +void mk_force_stop_cpu(struct mk_instance *instance, u64 phys_cpu) { safe_apic_wait_icr_idle(); apic_icr_write(APIC_DM_NMI | APIC_DEST_PHYSICAL, (u32)phys_cpu); diff --git a/drivers/firmware/arm_sdei.c b/drivers/firmware/arm_sdei.c index f39ed7ba3a385e..87bb476a07a47d 100644 --- a/drivers/firmware/arm_sdei.c +++ b/drivers/firmware/arm_sdei.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -370,6 +371,14 @@ static int sdei_platform_reset(void) int err; on_each_cpu(&_ipi_private_reset, NULL, true); + + /* + * Shared events and interrupt bindings are machine-wide, and belong + * to the kernel that spawned this one. + */ + if (mk_spawned()) + return 0; + err = sdei_api_shared_reset(); if (err) { pr_err("Failed to reset platform: %d\n", err); @@ -467,6 +476,16 @@ int sdei_event_disable(u32 event_num) return err; } +/* + * Success only means the event is now pending on @target_mpidr: firmware + * drops it if that CPU has not registered the event. + */ +int sdei_event_signal(u32 event_num, u64 target_mpidr) +{ + return invoke_sdei_fn(SDEI_1_0_FN_SDEI_EVENT_SIGNAL, event_num, + target_mpidr, 0, 0, 0, NULL); +} + static int sdei_api_event_unregister(u32 event_num) { return invoke_sdei_fn(SDEI_1_0_FN_SDEI_EVENT_UNREGISTER, event_num, 0, diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c index 38ca190d4a22d6..69f129178bd7e7 100644 --- a/drivers/firmware/psci/psci.c +++ b/drivers/firmware/psci/psci.c @@ -214,22 +214,46 @@ static int psci_0_2_cpu_off(u32 state) return __psci_cpu_off(PSCI_0_2_FN_CPU_OFF, state); } -static int __psci_cpu_on(u32 fn, unsigned long cpuid, unsigned long entry_point) +static int __psci_cpu_on(u32 fn, unsigned long cpuid, unsigned long entry_point, + unsigned long context_id) { int err; - err = invoke_psci_fn(fn, cpuid, entry_point, 0); + err = invoke_psci_fn(fn, cpuid, entry_point, context_id); return psci_to_linux_errno(err); } static int psci_0_1_cpu_on(unsigned long cpuid, unsigned long entry_point) { - return __psci_cpu_on(psci_0_1_function_ids.cpu_on, cpuid, entry_point); + return __psci_cpu_on(psci_0_1_function_ids.cpu_on, cpuid, entry_point, 0); } static int psci_0_2_cpu_on(unsigned long cpuid, unsigned long entry_point) { - return __psci_cpu_on(PSCI_FN_NATIVE(0_2, CPU_ON), cpuid, entry_point); + return __psci_cpu_on(PSCI_FN_NATIVE(0_2, CPU_ON), cpuid, entry_point, 0); +} + +/** + * psci_cpu_on_context - CPU_ON with a caller-chosen context ID + * @cpuid: MPIDR of the CPU to start + * @entry_point: physical address it starts executing at + * @context_id: value it finds in x0 (r0) there + * + * The secondary boot path has no use for the context ID and passes 0. + * Starting a CPU on a different kernel image does: the boot protocol + * wants the device tree address in that register. + */ +int psci_cpu_on_context(unsigned long cpuid, unsigned long entry_point, + unsigned long context_id) +{ + u32 fn = PSCI_FN_NATIVE(0_2, CPU_ON); + + if (!psci_ops.cpu_on) + return -EOPNOTSUPP; + if (psci_ops.cpu_on == psci_0_1_cpu_on) + fn = psci_0_1_function_ids.cpu_on; + + return __psci_cpu_on(fn, cpuid, entry_point, context_id); } static int __psci_migrate(u32 fn, unsigned long cpuid) diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile index 26aa3b6ec99fd4..3fb64393e51519 100644 --- a/drivers/irqchip/Makefile +++ b/drivers/irqchip/Makefile @@ -35,6 +35,7 @@ obj-$(CONFIG_ARM_GIC_V2M) += irq-gic-v2m.o obj-$(CONFIG_ARM_GIC_V3) += irq-gic-v3.o irq-gic-v3-mbi.o irq-gic-common.o obj-$(CONFIG_ARM_GIC_ITS_PARENT) += irq-gic-its-msi-parent.o obj-$(CONFIG_ARM_GIC_V3_ITS) += irq-gic-v3-its.o irq-gic-v4.o +obj-$(CONFIG_MULTIKERNEL_ITS_PROXY) += irq-gic-v3-its-mk.o obj-$(CONFIG_ARM_GIC_V3_ITS_FSL_MC) += irq-gic-v3-its-fsl-mc-msi.o obj-$(CONFIG_ARM_GIC_V5) += irq-gic-v5.o irq-gic-v5-irs.o irq-gic-v5-its.o \ irq-gic-v5-iwb.o diff --git a/drivers/irqchip/irq-gic-v3-its-mk.c b/drivers/irqchip/irq-gic-v3-its-mk.c new file mode 100644 index 00000000000000..53a1e2ec1fcc36 --- /dev/null +++ b/drivers/irqchip/irq-gic-v3-its-mk.c @@ -0,0 +1,272 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * MSI domain of a multikernel instance on a GICv3 with an ITS. + * + * The ITS belongs to the kernel that spawned this one, so this is what + * is left of an ITS driver when someone else runs the command queue: an + * MSI parent domain on top of the GIC's LPI range that asks the host to + * map each event (mk_msi_proxy_map()) and is told which LPI will arrive + * and where the device has to write the event. That address comes with + * every answer rather than with the device tree, because a machine can + * have several ITSs and only the host knows which one sits behind a + * device. The device is programmed from here as usual. + * + * The redistributors of this kernel's CPUs still run on the host's LPI + * tables, which the host enabled before it gave the CPUs away and which + * nothing can take back. So the host also owns the configuration bytes, + * and an LPI stays enabled from the moment it is mapped: masking is left + * to the MSI capability of the device. + */ + +#define pr_fmt(fmt) "ITS-mk: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include "irq-gic-its-msi-parent.h" +#include + +#define MK_ITS_COMPATIBLE "multikernel,gic-v3-its" + +struct mk_its_device { + u32 domain; + u32 rid; + u32 nvecs; + unsigned long *events; +}; + +struct mk_its_irq { + struct mk_its_device *dev; + u32 event; + phys_addr_t doorbell; +}; + +static void mk_its_nop(struct irq_data *d) +{ +} + +/* + * Called with the descriptor locked, and by a CPU on its way out that is + * turned off right after. Like the ITS driver's own, the move is complete + * when this returns: the host has sent the MOVI, which takes a pending + * interrupt along. + */ +static int mk_its_set_affinity(struct irq_data *d, const struct cpumask *mask, + bool force) +{ + const struct cpumask *now = irq_data_get_effective_affinity_mask(d); + struct mk_its_irq *irq = irq_data_get_irq_chip_data(d); + unsigned int cpu; + int ret; + + /* Stay where it is if that is allowed: a move is a round trip to the host */ + cpu = cpumask_first_and_and(now, mask, cpu_online_mask); + if (cpu < nr_cpu_ids) + return IRQ_SET_MASK_OK_DONE; + + cpu = cpumask_first_and(mask, cpu_online_mask); + if (cpu >= nr_cpu_ids) + return -EINVAL; + + ret = mk_msi_proxy_move(irq->dev->domain, irq->dev->rid, irq->event, + cpu_logical_map(cpu)); + if (ret) + return ret; + + irq_data_update_effective_affinity(d, cpumask_of(cpu)); + return IRQ_SET_MASK_OK_DONE; +} + +static void mk_its_compose_msi_msg(struct irq_data *d, struct msi_msg *msg) +{ + struct mk_its_irq *irq = irq_data_get_irq_chip_data(d); + + msg->address_lo = lower_32_bits(irq->doorbell); + msg->address_hi = upper_32_bits(irq->doorbell); + msg->data = irq->event; +} + +static struct irq_chip mk_its_chip = { + .name = "ITS-mk", + .irq_mask = mk_its_nop, + .irq_unmask = mk_its_nop, + .irq_eoi = irq_chip_eoi_parent, + .irq_set_affinity = mk_its_set_affinity, + .irq_compose_msi_msg = mk_its_compose_msi_msg, +}; + +static int mk_its_msi_prepare(struct irq_domain *domain, struct device *dev, + int nvec, msi_alloc_info_t *info) +{ + struct mk_its_device *mdev; + + if (!dev_is_pci(dev)) + return -EINVAL; + + mdev = kzalloc_obj(*mdev); + if (!mdev) + return -ENOMEM; + + mdev->nvecs = roundup_pow_of_two(nvec); + mdev->events = bitmap_zalloc(mdev->nvecs, GFP_KERNEL); + if (!mdev->events) { + kfree(mdev); + return -ENOMEM; + } + mdev->domain = pci_domain_nr(to_pci_dev(dev)->bus); + mdev->rid = pci_dev_id(to_pci_dev(dev)); + + info->scratchpad[0].ptr = mdev; + return 0; +} + +static void mk_its_msi_teardown(struct irq_domain *domain, msi_alloc_info_t *info) +{ + struct mk_its_device *mdev = info->scratchpad[0].ptr; + + bitmap_free(mdev->events); + kfree(mdev); +} + +static struct msi_domain_ops mk_its_msi_domain_ops = { + .msi_prepare = mk_its_msi_prepare, + .msi_teardown = mk_its_msi_teardown, +}; + +static int mk_its_alloc_one(struct irq_domain *domain, unsigned int virq, + struct mk_its_device *mdev, u32 event) +{ + struct irq_fwspec fwspec = { + .fwnode = domain->parent->fwnode, + .param_count = 3, + .param = { GIC_IRQ_TYPE_LPI, 0, IRQ_TYPE_EDGE_RISING }, + }; + unsigned int cpu = cpumask_first(cpu_online_mask); + struct mk_its_irq *irq; + struct irq_data *irqd; + int lpi, ret; + + irq = kzalloc_obj(*irq); + if (!irq) + return -ENOMEM; + irq->dev = mdev; + irq->event = event; + + lpi = mk_msi_proxy_map(mdev->domain, mdev->rid, event, + cpu_logical_map(cpu), &irq->doorbell); + if (lpi < 0) { + kfree(irq); + return lpi; + } + + fwspec.param[1] = lpi; + ret = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec); + if (ret) { + kfree(irq); + return ret; + } + + irq_domain_set_hwirq_and_chip(domain, virq, lpi, &mk_its_chip, irq); + irqd = irq_get_irq_data(virq); + irqd_set_single_target(irqd); + irq_data_update_effective_affinity(irqd, cpumask_of(cpu)); + return 0; +} + +static int mk_its_domain_alloc(struct irq_domain *domain, unsigned int virq, + unsigned int nr_irqs, void *args) +{ + msi_alloc_info_t *info = args; + struct mk_its_device *mdev = info->scratchpad[0].ptr; + unsigned int i; + int first, ret; + + first = bitmap_find_free_region(mdev->events, mdev->nvecs, + get_count_order(nr_irqs)); + if (first < 0) + return -ENOSPC; + + for (i = 0; i < nr_irqs; i++) { + ret = mk_its_alloc_one(domain, virq + i, mdev, first + i); + if (ret) + return ret; + } + return 0; +} + +/* The host keeps the mapping until this kernel is gone; the event is reusable */ +static void mk_its_domain_free(struct irq_domain *domain, unsigned int virq, + unsigned int nr_irqs) +{ + unsigned int i; + + for (i = 0; i < nr_irqs; i++) { + struct irq_data *d = irq_domain_get_irq_data(domain, virq + i); + struct mk_its_irq *irq = irq_data_get_irq_chip_data(d); + + if (irq) { + bitmap_clear(irq->dev->events, irq->event, 1); + kfree(irq); + } + irq_domain_reset_irq_data(d); + } + irq_domain_free_irqs_parent(domain, virq, nr_irqs); +} + +static const struct irq_domain_ops mk_its_domain_ops = { + .select = msi_lib_irq_domain_select, + .alloc = mk_its_domain_alloc, + .free = mk_its_domain_free, +}; + +static struct msi_domain_info mk_its_msi_info = { + .ops = &mk_its_msi_domain_ops, +}; + +/* + * No "interrupt-controller" property, so of_irq_init() does not come by. + * Any time after the GIC and before the first PCI host bridge will do. + */ +static int __init mk_its_init(void) +{ + struct irq_domain_info info = { + .ops = &mk_its_domain_ops, + .host_data = &mk_its_msi_info, + }; + struct device_node *np, *gic; + int ret = -ENODEV; + + np = of_find_compatible_node(NULL, NULL, MK_ITS_COMPATIBLE); + if (!np) + return 0; + + gic = of_irq_find_parent(np); + info.parent = gic ? irq_find_host(gic) : NULL; + of_node_put(gic); + + if (!info.parent) + goto out; + + info.fwnode = of_fwnode_handle(np); + ret = msi_create_parent_irq_domain(&info, &gic_v3_its_msi_parent_ops) ? + 0 : -ENOMEM; +out: + if (ret) + pr_err("%pOF: no MSI domain: %d\n", np, ret); + else + pr_info("MSIs through the host's ITS\n"); + of_node_put(np); + return ret; +} +arch_initcall(mk_its_init); diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 291d7668cc8daf..c20bf2ab5e634e 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -176,6 +176,7 @@ struct its_device { u32 nr_ites; u32 device_id; bool shared; + void *foreign; /* owner, when mapped for another kernel */ }; static struct { @@ -3601,6 +3602,12 @@ static int its_msi_prepare(struct irq_domain *domain, struct device *dev, mutex_lock(&its->dev_alloc_lock); its_dev = its_find_device(its, dev_id); + if (its_dev && its_dev->foreign) { + /* Mapped for another kernel, which is not an alias to share with */ + its_dev = NULL; + err = -EBUSY; + goto out; + } if (its_dev) { /* * We already have seen this ID, probably through @@ -3779,6 +3786,225 @@ static const struct irq_domain_ops its_domain_ops = { .deactivate = its_irq_domain_deactivate, }; +#ifdef CONFIG_MULTIKERNEL_ITS_PROXY +/* + * LPIs on behalf of another kernel. + * + * An ITS has one command queue, so a kernel that multikernel spawned on + * some of the machine's CPUs cannot map the MSIs of its devices itself. + * This kernel does it for it: the events of such a device are mapped to + * LPIs from this kernel's allocator and routed to the collection of a + * CPU the other kernel runs on. That collection is the one this kernel + * mapped for the CPU before it gave it away, and its redistributor still + * uses this kernel's LPI tables, which is why the configuration byte is + * written here as well. No Linux interrupt exists for these LPIs on this + * side: they never reach one of this kernel's CPU interfaces. + * + * Mapping sleeps. Moving a mapped event must not, because the other kernel + * waits for it with interrupts off, like this kernel's own its_set_affinity() + * waits for its MOVI; it may run in this kernel's interrupt handlers. + * its_foreign_lock is what such a move holds against the device going away + * under it. + * + * @owner keeps the devices of different kernels apart and names what + * its_foreign_release() tears down. The ITS is the one behind the device, + * named by the device's MSI domain: a machine has one per socket or more, + * each with DeviceIDs, collections and a doorbell of its own. + */ +static DEFINE_RAW_SPINLOCK(its_foreign_lock); + +static struct its_node *its_foreign_its(struct irq_domain *domain) +{ + if (!domain || domain->ops != &its_domain_ops) + return NULL; + return msi_get_domain_info(domain)->data; +} + +static void its_foreign_config(struct its_device *dev, u32 event, bool enable) +{ + u8 *cfg = gic_rdists->prop_table_va + + dev->event_map.lpi_base + event - 8192; + + *cfg = lpi_prop_prio | LPI_PROP_GROUP1 | (enable ? LPI_PROP_ENABLED : 0); + + if (gic_rdists->flags & RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING) + gic_flush_dcache_to_poc(cfg, sizeof(*cfg)); + else + dsb(ishst); + its_send_inv(dev, event); +} + +/* + * A collection is only mapped for a CPU that has been online here, and an + * ITS with erratum 23144 only reaches the CPUs of its own node. + */ +static struct its_collection *its_foreign_collection(struct its_node *its, + unsigned int cpu) +{ + if (cpu >= nr_cpu_ids || !cpu_possible(cpu) || + its->collections[cpu].col_id != cpu || + (cpu && !its->collections[cpu].target_address)) + return NULL; + + if ((its->flags & ITS_FLAGS_WORKAROUND_CAVIUM_23144) && + its->numa_node != NUMA_NO_NODE && cpu_to_node(cpu) != its->numa_node) + return NULL; + + return &its->collections[cpu]; +} + +/** + * its_foreign_move - Move a mapped event to another CPU of the other kernel + * @owner: the other kernel, as given to its_foreign_map() + * @domain: MSI domain of the device + * @dev_id: DeviceID as that ITS sees it + * @event: event to move, which its_foreign_map() has mapped + * @cpu: this kernel's number of the CPU to deliver to + * + * Returns once the ITS has moved the event, pending state included, so the + * CPU it came from may go away. Safe in any context. + * + * Returns 0 or a negative error code. + */ +int its_foreign_move(void *owner, struct irq_domain *domain, u32 dev_id, + u32 event, unsigned int cpu) +{ + struct its_node *its = its_foreign_its(domain); + struct its_collection *col; + struct its_device *dev; + + if (!its) + return -ENODEV; + col = its_foreign_collection(its, cpu); + if (!col) + return -EINVAL; + + guard(raw_spinlock_irqsave)(&its_foreign_lock); + + dev = its_find_device(its, dev_id); + if (!dev || dev->foreign != owner || event >= dev->event_map.nr_lpis || + !test_bit(event, dev->event_map.lpi_map)) + return -ENOENT; + + dev->event_map.col_map[event] = cpu; + its_send_movi(dev, col, event); + return 0; +} + +/** + * its_foreign_map - Route a device's event to a CPU of another kernel + * @owner: the other kernel, as the caller names it + * @domain: MSI domain of the device, which names the ITS behind it + * @dev_id: DeviceID as that ITS sees it + * @event: event (MSI data) to map + * @nvecs: events the device will use at most, fixed by the first call + * @cpu: this kernel's number of the CPU to deliver to + * @doorbell: set to the address the device has to write the event to + * + * Mapping an event that is mapped already moves it to @cpu. + * + * Returns the LPI, which is the INTID the other kernel takes on @cpu, + * or a negative error code. + */ +int its_foreign_map(void *owner, struct irq_domain *domain, u32 dev_id, + u32 event, u32 nvecs, unsigned int cpu, + phys_addr_t *doorbell) +{ + struct its_node *its = its_foreign_its(domain); + struct its_collection *col; + struct its_device *dev; + + if (!its) + return -ENODEV; + col = its_foreign_collection(its, cpu); + if (!col) + return -EINVAL; + + guard(mutex)(&its->dev_alloc_lock); + + dev = its_find_device(its, dev_id); + if (dev && dev->foreign != owner) + return -EBUSY; + if (!dev) { + dev = its_create_device(its, dev_id, + roundup_pow_of_two(max(nvecs, 1U)), true); + if (!dev) + return -ENOMEM; + dev->foreign = owner; + } + if (event >= dev->event_map.nr_lpis) + return -ERANGE; + + scoped_guard(raw_spinlock_irqsave, &its_foreign_lock) { + dev->event_map.col_map[event] = cpu; + if (test_and_set_bit(event, dev->event_map.lpi_map)) { + its_send_movi(dev, col, event); + } else { + its_send_mapti(dev, dev->event_map.lpi_base + event, + event); + its_foreign_config(dev, event, true); + } + } + *doorbell = its->get_msi_base(dev); + return dev->event_map.lpi_base + event; +} + +static void its_foreign_release_on(struct its_node *its, void *owner) +{ + struct its_device *dev, *tmp; + unsigned int event; + + guard(mutex)(&its->dev_alloc_lock); + + list_for_each_entry_safe(dev, tmp, &its->its_device_list, entry) { + if (dev->foreign != owner) + continue; + + /* No move finds the device from here on, and none is under way */ + scoped_guard(raw_spinlock_irqsave, &its_foreign_lock) + dev->foreign = NULL; + + for_each_set_bit(event, dev->event_map.lpi_map, + dev->event_map.nr_lpis) { + its_foreign_config(dev, event, false); + its_send_discard(dev, event); + } + + its_lpi_free(dev->event_map.lpi_map, dev->event_map.lpi_base, + dev->event_map.nr_lpis); + its_send_mapd(dev, 0); + its_free_device(dev); + } +} + +/** + * its_foreign_release - Drop everything mapped for another kernel + * @owner: the other kernel, as given to its_foreign_map() + * + * For when that kernel is gone or about to be started afresh, whether it + * shut down in order or not: nothing of it must keep raising LPIs. + */ +void its_foreign_release(void *owner) +{ + struct its_node *its; + + /* ITSs are only ever added, and that during boot */ + list_for_each_entry(its, &its_nodes, entry) + its_foreign_release_on(its, owner); +} + +/* + * The doorbell of the ITS behind a device with MSI domain @domain, which + * an IOMMU in front of the device has to let through. 0 if there is none. + */ +phys_addr_t its_foreign_doorbell(struct irq_domain *domain) +{ + struct its_node *its = its_foreign_its(domain); + + return its ? its->phys_base + GITS_TRANSLATER : 0; +} +#endif + /* * This is insane. * diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 20f13b686ab22f..f81446c59ad91f 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -50,6 +50,7 @@ static struct cpumask broken_rdists __read_mostly __maybe_unused; struct redist_region { void __iomem *redist_base; phys_addr_t phys_base; + resource_size_t size; bool single_redist; }; @@ -67,6 +68,7 @@ struct gic_chip_data { unsigned int ppi_nr; struct partition_affinity *parts; unsigned int nr_parts; + bool tenant; }; struct partition_affinity { @@ -920,6 +922,31 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs) __gic_handle_irq_from_irqson(regs); } +/* + * Tenant mode, for a kernel that shares the GIC with the kernel that + * spawned it ("multikernel,tenant" in the GIC node). + * + * Redistributors and CPU interfaces are per CPU, and a tenant drives + * those of its own CPUs like any other kernel. The distributor exists + * once: initialising it would disable, regroup and reroute every SPI + * of the machine, the host's included. A tenant leaves it as the host + * set it up and maps no SPIs; its devices interrupt with MSIs. + */ +static int __init gic_tenant_check_dist(void) +{ + u32 need = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A; + u32 val = readl_relaxed(gic_data.dist_base + GICD_CTLR); + + if ((val & need) != need) { + pr_err("Tenant of a distributor its owner left unusable (GICD_CTLR %#x)\n", + val); + return -ENXIO; + } + + pr_info("Tenant mode: the distributor is another kernel's\n"); + return 0; +} + static void __init gic_dist_init(void) { unsigned int i; @@ -1014,6 +1041,10 @@ static int gic_iterate_rdists(int (*fn)(struct redist_region *, void __iomem *)) if (typer & GICR_TYPER_VLPIS) ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */ } + /* A DT region may describe one frame without its Last bit set. */ + if (ptr - gic_data.redist_regions[i].redist_base >= + gic_data.redist_regions[i].size) + break; } while (!(typer & GICR_TYPER_LAST)); } @@ -1322,7 +1353,7 @@ static int gic_starting_cpu(unsigned int cpu) gic_cpu_sys_reg_enable(); gic_cpu_init(); - if (gic_dist_supports_lpis()) + if (gic_dist_supports_lpis() && !gic_data.tenant) its_cpu_init(); return 0; @@ -1401,6 +1432,32 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) isb(); } +/** + * gic_v3_send_sgi_to_mpidr - Send an SGI to a CPU named by its affinity + * @mpidr: MPIDR affinity of the target + * @sgi: SGI number + * + * gic_ipi_send_mask() reaches the CPUs of this kernel, by logical number. + * SGIs are delivered to a redistributor by affinity, though, so they + * cross to a CPU that another kernel image runs on just as well. + * + * Returns 0, or a negative error code if the SGI cannot be sent. + */ +int gic_v3_send_sgi_to_mpidr(u64 mpidr, unsigned int sgi) +{ + if (!gic_data.domain) + return -ENODEV; + if (sgi >= SGI_NR) + return -EINVAL; + if (MPIDR_AFFINITY_LEVEL(mpidr, 0) >= 16 && !gic_data.has_rss) + return -ERANGE; + + dsb(ishst); + gic_send_sgi(MPIDR_TO_SGI_CLUSTER_ID(mpidr), BIT(mpidr & 0xf), sgi); + isb(); + return 0; +} + static void __init gic_smp_init(void) { struct irq_fwspec sgi_fwspec = { @@ -1564,6 +1621,8 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq, case SPI_RANGE: case ESPI_RANGE: + if (gic_data.tenant) + return -EPERM; irq_domain_set_info(d, irq, hw, chip, d->host_data, handle_fasteoi_irq, NULL, NULL); irq_set_probe(irq); @@ -2028,7 +2087,8 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base, gic_data.has_rss = !!(typer & GICD_TYPER_RSS); - if (typer & GICD_TYPER_MBIS) { + /* MBIs are SPIs used as MSIs, and a tenant maps no SPIs */ + if ((typer & GICD_TYPER_MBIS) && !gic_data.tenant) { err = mbi_init(handle, gic_data.domain); if (err) pr_err("Failed to initialize MBIs\n"); @@ -2040,12 +2100,25 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base, gic_cpu_sys_reg_enable(); gic_prio_init(); - gic_dist_init(); + if (gic_data.tenant) { + err = gic_tenant_check_dist(); + if (err) + goto out_free; + } else { + gic_dist_init(); + } gic_cpu_init(); gic_enable_nmi_support(); gic_smp_init(); gic_cpu_pm_init(); + /* + * A tenant's redistributors keep the LPI tables the host enabled + * them with, and the host programs the ITSs for it. + */ + if (gic_data.tenant) + return 0; + if (gic_dist_supports_lpis()) { its_init(handle, &gic_data.rdists, gic_data.domain, dist_prio_irq); its_cpu_init(); @@ -2232,12 +2305,14 @@ static int __init gic_of_init(struct device_node *node, struct device_node *pare goto out_unmap_rdist; } rdist_regs[i].phys_base = res.start; + rdist_regs[i].size = resource_size(&res); } if (of_property_read_u64(node, "redistributor-stride", &redist_stride)) redist_stride = 0; gic_enable_of_quirks(node, gic_quirks, &gic_data); + gic_data.tenant = of_property_read_bool(node, "multikernel,tenant"); err = gic_init_bases(dist_phys_base, dist_base, rdist_regs, nr_redist_regions, redist_stride, &node->fwnode); @@ -2276,12 +2351,14 @@ static struct } acpi_data __initdata; static void __init -gic_acpi_register_redist(phys_addr_t phys_base, void __iomem *redist_base) +gic_acpi_register_redist(phys_addr_t phys_base, void __iomem *redist_base, + resource_size_t size) { static int count = 0; acpi_data.redist_regs[count].phys_base = phys_base; acpi_data.redist_regs[count].redist_base = redist_base; + acpi_data.redist_regs[count].size = size; acpi_data.redist_regs[count].single_redist = acpi_data.single_redist; count++; } @@ -2306,7 +2383,7 @@ gic_acpi_parse_madt_redist(union acpi_subtable_headers *header, gic_request_region(redist->base_address, redist->length, "GICR"); - gic_acpi_register_redist(redist->base_address, redist_base); + gic_acpi_register_redist(redist->base_address, redist_base, redist->length); return 0; } @@ -2348,7 +2425,7 @@ gic_acpi_parse_madt_gicc(union acpi_subtable_headers *header, (gicc->flags & ACPI_MADT_GICC_NON_COHERENT)) gic_data.rdists.flags |= RDIST_FLAGS_FORCE_NON_SHAREABLE; - gic_acpi_register_redist(gicc->gicr_base_address, redist_base); + gic_acpi_register_redist(gicc->gicr_base_address, redist_base, size); return 0; } diff --git a/drivers/pci/access.c b/drivers/pci/access.c index b123da16b63ba3..828f80f78f944f 100644 --- a/drivers/pci/access.c +++ b/drivers/pci/access.c @@ -259,6 +259,9 @@ int pci_user_write_config_##size \ if (PCI_##size##_BAD) \ return -EINVAL; \ \ + if (dev->foreign) \ + return -EPERM; \ + \ raw_spin_lock_irq(&pci_lock); \ if (unlikely(dev->block_cfg_access)) \ pci_wait_cfg(dev); \ @@ -592,6 +595,8 @@ int pci_write_config_byte(const struct pci_dev *dev, int where, u8 val) { if (pci_dev_is_disconnected(dev)) return PCIBIOS_DEVICE_NOT_FOUND; + if (dev->foreign) + return PCIBIOS_SUCCESSFUL; return pci_bus_write_config_byte(dev->bus, dev->devfn, where, val); } EXPORT_SYMBOL(pci_write_config_byte); @@ -600,6 +605,8 @@ int pci_write_config_word(const struct pci_dev *dev, int where, u16 val) { if (pci_dev_is_disconnected(dev)) return PCIBIOS_DEVICE_NOT_FOUND; + if (dev->foreign) + return PCIBIOS_SUCCESSFUL; return pci_bus_write_config_word(dev->bus, dev->devfn, where, val); } EXPORT_SYMBOL(pci_write_config_word); @@ -609,6 +616,8 @@ int pci_write_config_dword(const struct pci_dev *dev, int where, { if (pci_dev_is_disconnected(dev)) return PCIBIOS_DEVICE_NOT_FOUND; + if (dev->foreign) + return PCIBIOS_SUCCESSFUL; return pci_bus_write_config_dword(dev->bus, dev->devfn, where, val); } EXPORT_SYMBOL(pci_write_config_dword); diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 6c1ad1f542d9be..53b34c48d66ce6 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -368,7 +368,7 @@ void pci_bus_add_device(struct pci_dev *dev) */ pm_runtime_enable(&dev->dev); - if (!dn || of_device_is_available(dn)) + if ((!dn || of_device_is_available(dn)) && !dev->foreign) pci_dev_allow_binding(dev); device_initial_probe(&dev->dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 7661517ebdf0df..d6ca9646e32f6a 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2036,6 +2036,8 @@ int pci_setup_device(struct pci_dev *dev) err = pci_set_of_node(dev); if (err) return err; + /* Known before the first write: sizing the BARs is one */ + dev->foreign = mk_pci_foreign(dev); pci_set_acpi_fwnode(dev); pci_dev_assign_slot(dev); diff --git a/include/linux/arm_sdei.h b/include/linux/arm_sdei.h index f652a5028b5907..0810f44ef1ade9 100644 --- a/include/linux/arm_sdei.h +++ b/include/linux/arm_sdei.h @@ -48,11 +48,18 @@ int sdei_mask_local_cpu(void); int sdei_unmask_local_cpu(void); void __init acpi_sdei_init(void); void sdei_handler_abort(void); + +/* Only event 0, the software-signalled event, can be signalled */ +int sdei_event_signal(u32 event_num, u64 target_mpidr); #else static inline int sdei_mask_local_cpu(void) { return 0; } static inline int sdei_unmask_local_cpu(void) { return 0; } static inline void acpi_sdei_init(void) { } static inline void sdei_handler_abort(void) { } +static inline int sdei_event_signal(u32 event_num, u64 target_mpidr) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_ARM_SDE_INTERFACE */ diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h index 0225121f301380..e563e3686e94e3 100644 --- a/include/linux/irqchip/arm-gic-v3.h +++ b/include/linux/irqchip/arm-gic-v3.h @@ -638,6 +638,15 @@ struct irq_domain; struct fwnode_handle; int __init its_lpi_memreserve_init(void); int its_cpu_init(void); +int gic_v3_send_sgi_to_mpidr(u64 mpidr, unsigned int sgi); + +int its_foreign_map(void *owner, struct irq_domain *domain, u32 dev_id, + u32 event, u32 nvecs, unsigned int cpu, + phys_addr_t *doorbell); +int its_foreign_move(void *owner, struct irq_domain *domain, u32 dev_id, + u32 event, unsigned int cpu); +void its_foreign_release(void *owner); +phys_addr_t its_foreign_doorbell(struct irq_domain *domain); int its_init(struct fwnode_handle *handle, struct rdists *rdists, struct irq_domain *domain, u8 irq_prio); int mbi_init(struct fwnode_handle *fwnode, struct irq_domain *parent); diff --git a/include/linux/multikernel.h b/include/linux/multikernel.h index 5b1251687f32b9..8b6f599f362a2b 100644 --- a/include/linux/multikernel.h +++ b/include/linux/multikernel.h @@ -131,6 +131,15 @@ struct mk_shared_data { * the struct on re-exec. */ u8 parked[MK_PARKED_MAX]; + /* + * CPUs that firmware can stop out of band, indexed like parked[] + * and written by each CPU for itself: set once the architecture's + * firmware stop event can reach it, cleared before it stops being + * reachable. A force halt only asks firmware to stop the CPUs set + * here, because a request to any other CPU may leave it unable to + * take the stop interrupt as well. + */ + u8 fw_stop[MK_PARKED_MAX]; }; /* Function pointer type for IPI callbacks */ @@ -175,6 +184,41 @@ int multikernel_send_ipi_data(int instance_id, void *data, size_t data_size, uns void generic_multikernel_interrupt(void); +/* Drain the message ring where the doorbell interrupt cannot be taken */ +void mk_ipi_poll(void); + +struct mk_instance; +int multikernel_send_ipi_data_to(struct mk_instance *instance, void *data, + size_t data_size, unsigned long type); + +/* + * Host side: keep the DMA of an instance's PCI devices inside its memory + * while it runs, where the devices sit behind an IOMMU. + */ +#ifdef CONFIG_MULTIKERNEL_IOMMU +int mk_iommu_contain(struct mk_instance *instance); +void mk_iommu_release(struct mk_instance *instance); +int mk_iommu_map(struct mk_instance *instance, phys_addr_t start, size_t size); +void mk_iommu_unmap(struct mk_instance *instance, phys_addr_t start, size_t size); +#else +static inline int mk_iommu_contain(struct mk_instance *instance) +{ + return 0; +} +static inline void mk_iommu_release(struct mk_instance *instance) +{ +} +static inline int mk_iommu_map(struct mk_instance *instance, + phys_addr_t start, size_t size) +{ + return 0; +} +static inline void mk_iommu_unmap(struct mk_instance *instance, + phys_addr_t start, size_t size) +{ +} +#endif + /* Discard everything queued in this kernel's ring (instance re-spawn) */ void mk_ipi_ring_drop_pending(void); @@ -198,6 +242,10 @@ void mk_ipi_ring_drop_pending(void); #define MK_IO_IRQ_BALANCE (MK_MSG_IO + 2) #define MK_IO_IRQ_MASK (MK_MSG_IO + 3) #define MK_IO_IRQ_UNMASK (MK_MSG_IO + 4) +/* MSI routing through the host, where the arch needs it: payload is the arch's */ +#define MK_IO_MSI_MAP (MK_MSG_IO + 5) /* Route a device's MSI to one of my CPUs */ +#define MK_IO_MSI_ACK (MK_MSG_IO + 6) /* Answer: the interrupt number, or an error */ +#define MK_IO_MSI_MOVE (MK_MSG_IO + 7) /* Move a routed MSI to another of my CPUs */ /* Resource management subtypes */ #define MK_RES_CPU_ADD (MK_MSG_RESOURCE + 1) @@ -331,6 +379,15 @@ struct mk_pending_msg; int mk_send_message(int instance_id, u32 msg_type, u32 subtype, void *payload, u32 payload_len); +/* + * The same for a caller that holds a reference to @instance. Looking an + * instance up by ID sleeps; this does not, so it is the one to use with + * interrupts off. + */ +struct mk_instance; +int mk_send_message_to(struct mk_instance *instance, u32 msg_type, u32 subtype, + void *payload, u32 payload_len); + /** * mk_register_msg_handler - Register handler for specific message type * @msg_type: Message type to handle @@ -709,13 +766,14 @@ struct mk_instance { size_t ctrl_used; /* - * CPUs parked inside this instance's memory (its arch park area) - * rather than in the host pool. They must be confirmed parked before - * the image is rewritten and they pin the pool until they leave. - * Membership is per CPU because it changes one CPU at a time: - * hotplug moves CPUs in and out while the instance runs, and a - * stopped instance can gain CPUs that still sit in the host pool. - * Always empty where firmware holds parked CPUs. + * CPUs the host pool cannot reach yet: parked inside this instance's + * memory (its arch park area), or, where firmware holds parked CPUs, + * handed to this instance's kernel and not seen off since. They must + * be confirmed parked before the image is rewritten and they pin the + * pool until they leave. Membership is per CPU because it changes one + * CPU at a time: hotplug moves CPUs in and out while the instance + * runs, and a stopped instance can gain CPUs that still sit in the + * host pool. */ struct mk_cpu_set *cpus_on_slot; @@ -959,6 +1017,7 @@ void mk_kimage_free(struct kimage *image, void *virt_addr, size_t size); /* Device probe filtering against the instance's allowlist */ bool mk_pci_should_probe(struct pci_bus *bus, int devfn); +bool mk_pci_foreign(const struct pci_dev *pdev); bool mk_platform_device_allowed(const char *name, const char *hid); /* Early CPU registration from the manifest (spawn kernels) */ @@ -967,6 +1026,9 @@ void mk_register_cpus_from_manifest(void); /* Accept the manifest handed over at boot (spawn kernels) */ void mk_manifest_populate(phys_addr_t fdt_phys, u64 fdt_len); +/* True in a kernel another kernel spawned, i.e. one that booted with a manifest */ +bool mk_spawned(void); + /* Build the manifest for a spawn (host, kexec path) */ int mk_manifest_finalize(struct kimage *image); #else @@ -1011,6 +1073,10 @@ static inline bool mk_pci_should_probe(struct pci_bus *bus, int devfn) { return true; } +static inline bool mk_pci_foreign(const struct pci_dev *pdev) +{ + return false; +} static inline bool mk_platform_device_allowed(const char *name, const char *hid) { return true; @@ -1021,6 +1087,10 @@ static inline void mk_register_cpus_from_manifest(void) static inline void mk_manifest_populate(phys_addr_t fdt_phys, u64 fdt_len) { } +static inline bool mk_spawned(void) +{ + return false; +} #endif /** @@ -1119,6 +1189,8 @@ void *mk_instance_ctrl_alloc(struct mk_instance *instance, size_t size, * - CONFIG_ARCH_HAS_MK_HOST_PARK and the pool park functions declared * with the pool chunk API above, for architectures that park CPUs in * software rather than in firmware. + * - CONFIG_ARCH_HAS_MK_MSI_PROXY and the two functions below, for + * architectures where an instance cannot route its own MSIs. */ /* Doorbell for the message ring: IPI a CPU owned by another kernel */ @@ -1135,7 +1207,7 @@ void __noreturn mk_enter_pool_state(void *info); * pseudo-NMI on arm64). Registration may fail where the architecture * has nothing suitable; graceful shutdown must keep working without it. */ -void mk_force_stop_cpu(mk_phys_cpu_t phys_cpu); +void mk_force_stop_cpu(struct mk_instance *instance, mk_phys_cpu_t phys_cpu); /* * Build the boot state for @instance from the loaded @image and start @@ -1166,4 +1238,25 @@ int mk_repark_instance_to_host(struct mk_instance *instance); int mk_repark_cpu_to_instance(struct mk_instance *instance, mk_phys_cpu_t phys_cpu); int mk_repark_cpu_to_host(struct mk_instance *instance, mk_phys_cpu_t phys_cpu); +/* + * MSIs that the host has to route for an instance (a GICv3 ITS has one + * command queue). The generic code only needs to know when to let go and + * what an IOMMU in front of a device has to let through. + */ +#ifdef CONFIG_ARCH_HAS_MK_MSI_PROXY +/* Drop every MSI routed for @instance: it halted, or is about to be started */ +void mk_arch_msi_release(struct mk_instance *instance); +/* The address @pdev writes its MSIs to when an instance owns it, 0 if none */ +phys_addr_t mk_arch_msi_doorbell(struct pci_dev *pdev); +#else +static inline void mk_arch_msi_release(struct mk_instance *instance) +{ +} + +static inline phys_addr_t mk_arch_msi_doorbell(struct pci_dev *pdev) +{ + return 0; +} +#endif + #endif /* _LINUX_MULTIKERNEL_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index 1c270f1d512301..79761d6895a440 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -490,6 +490,13 @@ struct pci_dev { * itself internal but devices downstream from it are external. */ unsigned int external_facing:1; + /* + * Seen by this kernel, owned by another: a bridge above the devices + * of a multikernel spawn, which the kernel that spawned it keeps + * driving. No driver binds to it and config writes are dropped, as + * if it were read-only. Its owner configured it. + */ + unsigned int foreign:1; unsigned int broken_intx_masking:1; /* INTx masking can't be used */ unsigned int io_window_1k:1; /* Intel bridge 1K I/O windows */ unsigned int irq_managed:1; diff --git a/include/linux/psci.h b/include/linux/psci.h index 4ca0060a3fc42b..5996cdd148771e 100644 --- a/include/linux/psci.h +++ b/include/linux/psci.h @@ -43,6 +43,9 @@ struct psci_0_1_function_ids { struct psci_0_1_function_ids get_psci_0_1_function_ids(void); +int psci_cpu_on_context(unsigned long cpuid, unsigned long entry_point, + unsigned long context_id); + #if defined(CONFIG_ARM_PSCI_FW) int __init psci_dt_init(void); #else diff --git a/include/uapi/linux/arm_sdei.h b/include/uapi/linux/arm_sdei.h index af0630ba5437d4..22eb61612673dc 100644 --- a/include/uapi/linux/arm_sdei.h +++ b/include/uapi/linux/arm_sdei.h @@ -22,6 +22,7 @@ #define SDEI_1_0_FN_SDEI_PE_UNMASK SDEI_1_0_FN(0x0C) #define SDEI_1_0_FN_SDEI_INTERRUPT_BIND SDEI_1_0_FN(0x0D) #define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE SDEI_1_0_FN(0x0E) +#define SDEI_1_0_FN_SDEI_EVENT_SIGNAL SDEI_1_0_FN(0x0F) #define SDEI_1_0_FN_SDEI_PRIVATE_RESET SDEI_1_0_FN(0x11) #define SDEI_1_0_FN_SDEI_SHARED_RESET SDEI_1_0_FN(0x12) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index 3641b022311248..d5f42504eddf27 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -1754,7 +1754,19 @@ int multikernel_kexec_by_id(int mk_id) */ mk_ipi_ring_drop_pending(); + /* A previous run that died without a word left its MSIs routed */ + mk_arch_msi_release(instance); + + rc = mk_iommu_contain(instance); + if (rc) { + pr_err("Instance %d: devices not confined to its memory: %d\n", + mk_id, rc); + goto unlock; + } + rc = mk_arch_spawn_instance(mk_image, instance, cpu); + if (rc) + mk_iommu_release(instance); if (rc == 0) { rc = mk_instance_set_kexec_active(mk_image->mk_id); if (rc) diff --git a/kernel/multikernel/Kconfig b/kernel/multikernel/Kconfig index cec1ea32d83361..fe4e1ca461831c 100644 --- a/kernel/multikernel/Kconfig +++ b/kernel/multikernel/Kconfig @@ -10,6 +10,10 @@ config ARCH_SUPPORTS_MULTIKERNEL config ARCH_HAS_MK_HOST_PARK bool +# The arch routes the MSIs of an instance's devices on the instance's behalf +config ARCH_HAS_MK_MSI_PROXY + bool + config MULTIKERNEL bool "Multikernel support" depends on ARCH_SUPPORTS_MULTIKERNEL @@ -33,3 +37,9 @@ config MULTIKERNEL boot channel, describing its resources and message rings If unsure, say N. + + +# Confine the DMA of an instance's PCI devices to its memory. x86 runs +# them without an IOMMU so far. +config MULTIKERNEL_IOMMU + def_bool MULTIKERNEL && IOMMU_API && PCI && ARM64 diff --git a/kernel/multikernel/Makefile b/kernel/multikernel/Makefile index cf2e1391038ec7..725b4b2db8ce4d 100644 --- a/kernel/multikernel/Makefile +++ b/kernel/multikernel/Makefile @@ -5,6 +5,7 @@ obj-y += core.o cpuset.o mem.o contig.o kernfs.o dts.o instance_dt.o manifest.o ipi.o messaging.o overlay.o hotplug.o baseline.o obj-$(CONFIG_OF) += of_devices.o +obj-$(CONFIG_MULTIKERNEL_IOMMU) += iommu.o # DMA-BUF heap for multikernel memory allocation obj-$(CONFIG_DMABUF_HEAPS) += dma_heap.o diff --git a/kernel/multikernel/core.c b/kernel/multikernel/core.c index 4554a412bd2800..1486538d0bb973 100644 --- a/kernel/multikernel/core.c +++ b/kernel/multikernel/core.c @@ -1605,6 +1605,8 @@ static void mk_instance_settle_halted(struct mk_instance *instance) { pr_info("Instance %d (%s) halted, CPUs parking in pool\n", instance->id, instance->name); + mk_arch_msi_release(instance); + mk_iommu_release(instance); mk_instance_set_state(instance, MK_STATE_LOADED); } @@ -1749,6 +1751,8 @@ int multikernel_halt_by_id(int mk_id) pr_warn("Multikernel instance %d halted with CPUs unaccounted for\n", mk_id); + mk_arch_msi_release(instance); + mk_iommu_release(instance); mk_instance_set_state(instance, MK_STATE_LOADED); pr_info("Multikernel instance %d halted (graceful)\n", mk_id); } @@ -1942,7 +1946,7 @@ int multikernel_force_halt_by_id(int mk_id) pr_err("Failed to arm force-halt marker: %d (sending NMI anyway)\n", ret); mk_cpu_set_for_each(i, phys_cpu, targets) { - mk_force_stop_cpu(phys_cpu); + mk_force_stop_cpu(instance, phys_cpu); cpu_count++; } diff --git a/kernel/multikernel/dts.c b/kernel/multikernel/dts.c index a62aae701f8ab9..99a514482b0daf 100644 --- a/kernel/multikernel/dts.c +++ b/kernel/multikernel/dts.c @@ -20,6 +20,7 @@ #include #include #include +#include #if defined(CONFIG_X86) && defined(CONFIG_PCI_MMCONFIG) #include #endif @@ -1224,6 +1225,18 @@ static u64 mk_dt_bridge_ecam(struct pci_bus *root, u64 *size) *size = (u64)(root->busn_res.end - root->busn_res.start + 1) << 20; return cfg->address + ((u64)root->busn_res.start << 20); +#elif defined(CONFIG_PCI_ECAM) + struct pci_config_window *cfg = root->sysdata; + + /* Only a bridge on the generic ECAM accessors keeps its window there */ + if (root->ops != &pci_generic_ecam_ops.pci_ops || + cfg->busr.start > root->busn_res.start || + cfg->busr.end < root->busn_res.end) + return 0; + + *size = (u64)(root->busn_res.end - root->busn_res.start + 1) << 20; + return cfg->res.start + + ((u64)(root->busn_res.start - cfg->busr.start) << 20); #else return 0; #endif @@ -1281,7 +1294,7 @@ static int mk_dt_emit_one_host_bridge(struct mk_instance *instance, resource_list_for_each_entry(entry, &bridge->windows) { const struct resource *res = entry->res; - u64 pci_addr, size; + u64 pci_addr, cpu_addr, size; if (resource_type(res) != IORESOURCE_IO && resource_type(res) != IORESOURCE_MEM) @@ -1295,11 +1308,21 @@ static int mk_dt_emit_one_host_bridge(struct mk_instance *instance, pci_addr = res->start - entry->offset; size = resource_size(res); + + /* + * Where I/O space is memory mapped, an I/O resource holds + * logical port numbers, not the CPU address of the window. + */ + cpu_addr = res->start; + if (resource_type(res) == IORESOURCE_IO && + pci_pio_to_address(res->start) != (phys_addr_t)OF_BAD_ADDR) + cpu_addr = pci_pio_to_address(res->start); + ranges[cells++] = cpu_to_fdt32(mk_dt_window_flags(res)); ranges[cells++] = cpu_to_fdt32(upper_32_bits(pci_addr)); ranges[cells++] = cpu_to_fdt32(lower_32_bits(pci_addr)); - ranges[cells++] = cpu_to_fdt32(upper_32_bits(res->start)); - ranges[cells++] = cpu_to_fdt32(lower_32_bits(res->start)); + ranges[cells++] = cpu_to_fdt32(upper_32_bits(cpu_addr)); + ranges[cells++] = cpu_to_fdt32(lower_32_bits(cpu_addr)); ranges[cells++] = cpu_to_fdt32(upper_32_bits(size)); ranges[cells++] = cpu_to_fdt32(lower_32_bits(size)); } diff --git a/kernel/multikernel/hotplug.c b/kernel/multikernel/hotplug.c index f393b368da149a..3f84e20f40defc 100644 --- a/kernel/multikernel/hotplug.c +++ b/kernel/multikernel/hotplug.c @@ -1455,16 +1455,29 @@ int mk_send_mem_add(int instance_id, u64 start_pfn, u64 nr_pages, goto out; } + /* Its devices must reach the memory by the time the instance uses it */ + ret = mk_iommu_map(target_instance, PFN_PHYS(start_pfn), + PFN_PHYS(nr_pages)); + if (ret < 0) { + mk_msg_pending_wait(pending, 0); + goto out; + } + ret = mk_send_message(instance_id, MK_MSG_RESOURCE, MK_RES_MEM_ADD, &payload, sizeof(payload)); if (ret < 0) { mk_msg_pending_wait(pending, 0); /* Immediate cleanup */ + mk_iommu_unmap(target_instance, PFN_PHYS(start_pfn), + PFN_PHYS(nr_pages)); goto out; } ret = mk_msg_pending_wait(pending, 10000); - if (ret < 0) + if (ret < 0) { + mk_iommu_unmap(target_instance, PFN_PHYS(start_pfn), + PFN_PHYS(nr_pages)); goto out; + } ret = mk_instance_add_memory_region(target_instance, PFN_PHYS(nr_pages), (int)numa_node); @@ -1541,6 +1554,7 @@ int mk_send_mem_remove(int instance_id, u64 start_pfn, u64 nr_pages) if (ret < 0) goto abort; + mk_iommu_unmap(target_instance, PFN_PHYS(start_pfn), PFN_PHYS(nr_pages)); mk_instance_finish_memory_removal(target_instance, PFN_PHYS(start_pfn), PFN_PHYS(nr_pages)); goto out; diff --git a/kernel/multikernel/instance_dt.c b/kernel/multikernel/instance_dt.c index 8503733372da18..3524179d043644 100644 --- a/kernel/multikernel/instance_dt.c +++ b/kernel/multikernel/instance_dt.c @@ -550,6 +550,27 @@ bool mk_pci_should_probe(struct pci_bus *bus, int devfn) } EXPORT_SYMBOL_GPL(mk_pci_should_probe); +/** + * mk_pci_foreign - Whether a PCI function this kernel sees is not its own + * @pdev: device being set up, its node attached + * + * A spawn kernel has to see the bridges above its devices, or it could not + * reach the buses behind them, but seeing is all. A port on the way down + * still belongs to the kernel that spawned this one: that kernel has a + * driver bound to it, and other kernels' devices may sit below it. In its + * tree a spawn tells the two apart by what the node says: one that names a + * device was given to it, a bridge on the way is described without one. + * + * Returns: true if the PCI core must leave the function alone + */ +bool mk_pci_foreign(const struct pci_dev *pdev) +{ + const struct device_node *np = pdev->dev.of_node; + + return mk_manifest_phys() && !(np && of_property_present(np, "vendor-id")); +} +EXPORT_SYMBOL_GPL(mk_pci_foreign); + /* * A netdev's alias is its interface name, the one the device had in the * kernel that gave it away, found from the node its device is bound to, diff --git a/kernel/multikernel/iommu.c b/kernel/multikernel/iommu.c new file mode 100644 index 00000000000000..e66e22b379f588 --- /dev/null +++ b/kernel/multikernel/iommu.c @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * DMA containment for the PCI devices of an instance. + * + * An instance drives its devices without an IOMMU of its own: the IOMMU + * stays with the host, the instance tree does not mention it, and the + * instance hands physical addresses to its devices. Behind a translating + * IOMMU that DMA faults, and behind a bypassing one a buggy driver in the + * instance can reach all of the host. + * + * So while an instance runs, each of its devices is attached to a domain + * of its own that maps the instance's memory grant onto itself, and with + * it the MSI doorbell of the ITS behind that device. The device works + * untranslated, and cannot reach anything else. The domain follows the + * grant as memory is added and removed, and goes away when the instance + * halts, which returns the device to the host's default domain for a host + * driver to bind. + * + * A device without an IOMMU is left alone: its DMA is unrestricted, as + * it always was. + */ + +#define pr_fmt(fmt) "mk_iommu: " fmt + +#include +#include +#include +#include +#include + +struct mk_iommu_dev { + struct list_head list; + struct mk_instance *instance; + struct pci_dev *pdev; + struct iommu_domain *domain; +}; + +static LIST_HEAD(mk_iommu_devs); +static DEFINE_MUTEX(mk_iommu_lock); + +static int mk_iommu_prot(struct mk_iommu_dev *d) +{ + return IOMMU_READ | IOMMU_WRITE | + (dev_is_dma_coherent(&d->pdev->dev) ? IOMMU_CACHE : 0); +} + +static void mk_iommu_free_dev(struct mk_iommu_dev *d) +{ + if (d->domain) { + iommu_detach_device(d->domain, &d->pdev->dev); + iommu_domain_free(d->domain); + } + iommu_device_release_dma_owner(&d->pdev->dev); + pci_dev_put(d->pdev); + list_del(&d->list); + kfree(d); +} + +static int mk_iommu_contain_dev(struct mk_instance *instance, + struct pci_dev *pdev) +{ + phys_addr_t doorbell = mk_arch_msi_doorbell(pdev); + struct mk_memory_region *region; + struct mk_iommu_dev *d; + int ret; + + d = kzalloc_obj(*d); + if (!d) + return -ENOMEM; + d->instance = instance; + d->pdev = pci_dev_get(pdev); + list_add(&d->list, &mk_iommu_devs); + + /* Fails while a host driver is bound, which must not share the device */ + ret = iommu_device_claim_dma_owner(&pdev->dev, instance); + if (ret) { + pci_dev_put(d->pdev); + list_del(&d->list); + kfree(d); + return ret; + } + + d->domain = iommu_paging_domain_alloc(&pdev->dev); + if (IS_ERR(d->domain)) { + ret = PTR_ERR(d->domain); + d->domain = NULL; + goto err; + } + + list_for_each_entry(region, &instance->memory_regions, list) { + ret = iommu_map(d->domain, region->res.start, region->res.start, + resource_size(®ion->res), mk_iommu_prot(d), + GFP_KERNEL); + if (ret) + goto err_domain; + } + + if (doorbell) { + ret = iommu_map(d->domain, doorbell & PAGE_MASK, + doorbell & PAGE_MASK, PAGE_SIZE, + IOMMU_WRITE | IOMMU_MMIO, GFP_KERNEL); + if (ret) + goto err_domain; + } + + ret = iommu_attach_device(d->domain, &pdev->dev); + if (!ret) + return 0; + +err_domain: + iommu_domain_free(d->domain); + d->domain = NULL; +err: + mk_iommu_free_dev(d); + return ret; +} + +/** + * mk_iommu_contain - Confine the DMA of an instance's devices to its memory + * @instance: Instance about to be started + * + * Returns 0, or a negative error code, in which case nothing is attached + * and the instance must not be started: its devices would either fault + * or roam. + */ +int mk_iommu_contain(struct mk_instance *instance) +{ + struct mk_pci_device *mine; + int ret = 0; + + mk_iommu_release(instance); + + if (!instance->pci_devices_valid) + return 0; + + mutex_lock(&mk_iommu_lock); + list_for_each_entry(mine, &instance->pci_devices, list) { + struct pci_dev *pdev; + + pdev = pci_get_domain_bus_and_slot(mine->domain, mine->bus, + PCI_DEVFN(mine->slot, mine->func)); + if (!pdev) + continue; + + if (device_iommu_mapped(&pdev->dev)) + ret = mk_iommu_contain_dev(instance, pdev); + else + pr_info_once("%s has no IOMMU, DMA of instance devices is unrestricted\n", + pci_name(pdev)); + if (ret) + pci_err(pdev, "DMA not confined to instance %d: %d\n", + instance->id, ret); + pci_dev_put(pdev); + if (ret) + break; + } + mutex_unlock(&mk_iommu_lock); + + if (ret) + mk_iommu_release(instance); + return ret; +} + +/** + * mk_iommu_release - Return an instance's devices to the host's domain + * @instance: Instance that has halted, or is about to be started afresh + */ +void mk_iommu_release(struct mk_instance *instance) +{ + struct mk_iommu_dev *d, *tmp; + + mutex_lock(&mk_iommu_lock); + list_for_each_entry_safe(d, tmp, &mk_iommu_devs, list) { + if (d->instance == instance) + mk_iommu_free_dev(d); + } + mutex_unlock(&mk_iommu_lock); +} + +/** + * mk_iommu_map - Let a running instance's devices reach memory it gains + * @instance: Instance the memory is being added to + * @start: Physical address of the range + * @size: Size of the range + * + * To be called before the instance is told about the memory. + */ +int mk_iommu_map(struct mk_instance *instance, phys_addr_t start, size_t size) +{ + struct mk_iommu_dev *d; + int ret = 0; + + mutex_lock(&mk_iommu_lock); + list_for_each_entry(d, &mk_iommu_devs, list) { + if (d->instance != instance) + continue; + ret = iommu_map(d->domain, start, start, size, mk_iommu_prot(d), + GFP_KERNEL); + if (ret) + break; + } + mutex_unlock(&mk_iommu_lock); + + if (ret) + mk_iommu_unmap(instance, start, size); + return ret; +} + +/** + * mk_iommu_unmap - Cut a running instance's devices off memory it lost + * @instance: Instance the memory was removed from + * @start: Physical address of the range + * @size: Size of the range + */ +void mk_iommu_unmap(struct mk_instance *instance, phys_addr_t start, size_t size) +{ + struct mk_iommu_dev *d; + + mutex_lock(&mk_iommu_lock); + list_for_each_entry(d, &mk_iommu_devs, list) { + if (d->instance == instance) + iommu_unmap(d->domain, start, size); + } + mutex_unlock(&mk_iommu_lock); +} diff --git a/kernel/multikernel/ipi.c b/kernel/multikernel/ipi.c index d7e9739a448dff..2720ee66ef5983 100644 --- a/kernel/multikernel/ipi.c +++ b/kernel/multikernel/ipi.c @@ -18,6 +18,8 @@ static struct mk_ipi_handler *mk_handlers; static raw_spinlock_t mk_handlers_lock = __RAW_SPIN_LOCK_UNLOCKED(mk_handlers_lock); +static DEFINE_RAW_SPINLOCK(mk_drain_lock); + static void mk_ipi_drain_ring(void); /* @@ -156,25 +158,39 @@ int mk_arm_force_halt(struct mk_instance *instance) */ int multikernel_send_ipi_data(int instance_id, void *data, size_t data_size, unsigned long type) { - struct mk_ipi_data *slot; struct mk_instance *instance = mk_instance_find(instance_id); + int ret; + + if (!instance) + return -EINVAL; + + ret = multikernel_send_ipi_data_to(instance, data, data_size, type); + mk_instance_put(instance); + return ret; +} + +/* + * The same for a caller that holds a reference to @instance. Looking an + * instance up by ID takes a mutex; this does not sleep, so it is the one + * to use with interrupts off. + */ +int multikernel_send_ipi_data_to(struct mk_instance *instance, void *data, + size_t data_size, unsigned long type) +{ struct mk_shared_data *ipi_data; + struct mk_ipi_data *slot; + int instance_id = instance->id; unsigned int head, next_head, tail; mk_phys_cpu_t target; - if (!instance) + if (data_size > MK_MAX_DATA_SIZE) return -EINVAL; - if (data_size > MK_MAX_DATA_SIZE) { - mk_instance_put(instance); - return -EINVAL; - } target = instance->ipi_target; if (target == MK_PHYS_CPU_INVALID) target = mk_cpu_set_first(instance->cpus); if (target == MK_PHYS_CPU_INVALID) { pr_err("Instance %d has no CPU to receive the IPI\n", instance_id); - mk_instance_put(instance); return -ENODEV; } @@ -182,7 +198,6 @@ int multikernel_send_ipi_data(int instance_id, void *data, size_t data_size, uns ipi_data = READ_ONCE(instance->ipi_data); if (!ipi_data) { pr_err("Multikernel IPI buffer not available for instance %d\n", instance_id); - mk_instance_put(instance); return -ENODEV; } @@ -202,7 +217,6 @@ int multikernel_send_ipi_data(int instance_id, void *data, size_t data_size, uns printk_deferred(KERN_WARNING "multikernel: IPI ring full for instance %d (head=%u, tail=%u)\n", instance_id, head, tail); - mk_instance_put(instance); return -ENOSPC; } @@ -227,8 +241,6 @@ int multikernel_send_ipi_data(int instance_id, void *data, size_t data_size, uns smp_store_release(&slot->data_size, data_size); mk_arch_send_ipi(target); - - mk_instance_put(instance); return 0; } @@ -310,6 +322,8 @@ static void mk_ipi_drain_ring(void) */ static void multikernel_interrupt_handler(void) { + unsigned long flags; + if (!mk_self || !mk_self->ipi_data) return; @@ -320,8 +334,28 @@ static void multikernel_interrupt_handler(void) * instance, so if it is ever left pending - its self-IPI lost while * the CPU was bringing its APIC up, say - every later queue attempt * is a no-op and the ring never drains again. + * + * The ring has one consumer at a time. That used to be implied by + * the doorbell going to a single CPU; mk_ipi_poll() is a second way + * in, so it is a lock now. Waiting for it rather than giving up + * matters: a poller that is just leaving may not look again. */ + raw_spin_lock_irqsave(&mk_drain_lock, flags); mk_ipi_drain_ring(); + raw_spin_unlock_irqrestore(&mk_drain_lock, flags); +} + +/** + * mk_ipi_poll - Drain the message ring without waiting for the doorbell + * + * For a caller that waits for an answer where the doorbell interrupt + * cannot be taken: with interrupts off on the doorbell CPU, or while + * stop_machine() holds every CPU. Message handlers run in the caller's + * context, which they are fit for, as they normally run in the interrupt. + */ +void mk_ipi_poll(void) +{ + multikernel_interrupt_handler(); } /** diff --git a/kernel/multikernel/manifest.c b/kernel/multikernel/manifest.c index 5220ac069e1f3f..23ddd99ca20653 100644 --- a/kernel/multikernel/manifest.c +++ b/kernel/multikernel/manifest.c @@ -34,6 +34,11 @@ phys_addr_t mk_manifest_phys(void) return mk_manifest_fdt_phys; } +bool mk_spawned(void) +{ + return mk_manifest_fdt_phys != 0; +} + /** * mk_manifest_populate() - Accept the manifest handed over at boot * @fdt_phys: Physical address of the manifest FDT diff --git a/kernel/multikernel/messaging.c b/kernel/multikernel/messaging.c index 2cd2b7682f8d3e..210f1d3d88618e 100644 --- a/kernel/multikernel/messaging.c +++ b/kernel/multikernel/messaging.c @@ -190,8 +190,9 @@ int mk_msg_pending_wait(struct mk_pending_msg *pending, unsigned long timeout_ms * * Returns 0 on success, negative error code on failure */ -int mk_send_message(int instance_id, u32 msg_type, u32 subtype, - void *payload, u32 payload_len) +static int mk_build_and_send(int instance_id, struct mk_instance *instance, + u32 msg_type, u32 subtype, void *payload, + u32 payload_len) { struct mk_message *msg; size_t total_size; @@ -223,7 +224,12 @@ int mk_send_message(int instance_id, u32 msg_type, u32 subtype, memcpy(msg->payload, payload, payload_len); /* Send via IPI using the message type as IPI type */ - ret = multikernel_send_ipi_data(instance_id, msg, total_size, msg_type); + if (instance) + ret = multikernel_send_ipi_data_to(instance, msg, total_size, + msg_type); + else + ret = multikernel_send_ipi_data(instance_id, msg, total_size, + msg_type); /* Clean up temporary buffer */ kfree(msg); @@ -238,8 +244,23 @@ int mk_send_message(int instance_id, u32 msg_type, u32 subtype, return 0; } + +int mk_send_message(int instance_id, u32 msg_type, u32 subtype, + void *payload, u32 payload_len) +{ + return mk_build_and_send(instance_id, NULL, msg_type, subtype, payload, + payload_len); +} EXPORT_SYMBOL(mk_send_message); +int mk_send_message_to(struct mk_instance *instance, u32 msg_type, u32 subtype, + void *payload, u32 payload_len) +{ + return mk_build_and_send(instance->id, instance, msg_type, subtype, + payload, payload_len); +} +EXPORT_SYMBOL(mk_send_message_to); + /** * mk_register_msg_handler - Register handler for specific message type * @msg_type: Message type to handle