Repository navigation
Description
Work has begun to provide support for WinUI 3.0. This support is expected to manifest in a way similar to the CppWinRT tool by way of a new source generation tool (e.g. CsWinRT). In order to support this new tool, APIs for integrating and coordinating with the runtime object lifetime are necessary.
Rationale and Usage
The below API surface provides a way for a third party tool to generate what are colloquially known as Runtime Callable Wrappers (RCW) and COM Callable Wrappers (CCW) in a way that allows safe interaction with managed object lifetime and identity.
A specific example of the need for lifetime coordination is in WinRT scenarios involving UI (e.g. WinUI 3.0) via the IReferenceTrackerManager interface.
Goals:
- Enable source generation of interop code in WinRT/WinUI scenarios.
- An API that generally aligns with how existing 3rd party source generators work (e.g. SharpGenTools).
- Limit exposing APIs that manage lifetime in a micro way (e.g. avoid GC hooks at dangerous times).
- Semantics in .NET Framework/.NET Core for WinRT scenarios should be able to be replicated.
- Provide a mechanism for 3rd parties to be able to support Reference Tracker scenarios.
- Ensure AOT scenarios are considered.
Non-Goals:
- Replace the existing built-in RCW/CCW infrastructure.
- Change anything related to P/Invoke Interop.
- Hide WinRT and/or COM concepts.
Outstanding questions:
- Implement new COM interop API for RCW/CCW creation/management #32091 (comment)
- Implement new COM interop API for RCW/CCW creation/management #32091 (comment)
Proposed API
namespace System.Runtime
{
public static partial class RuntimeHelpers
{
/// <summary>
/// Allocate memory that is associated with the <paramref name="type"/> and
/// will be freed if and when the <see cref="System.Type"/> is unloaded.
/// </summary>
/// <param name="type">Type associated with the allocated memory.</param>
/// <param name="size">Amount of memory in bytes to allocate.</param>
/// <returns>The allocated memory</returns>
public static IntPtr AllocateTypeAssociatedMemory(Type type, int size);
}
}
namespace System.Runtime.InteropServices
{
/// <summary>
/// Enumeration of flags for <see cref="ComWrappers.GetOrCreateComInterfaceForObject(object, CreateComInterfaceFlags)"/>.
/// </summary>
[Flags]
public enum CreateComInterfaceFlags
{
None = 0,
/// <summary>
/// The caller will provide an IUnknown Vtable.
/// </summary>
/// <remarks>
/// This is useful in scenarios when the caller has no need to rely on an IUnknown instance
/// that is used when running managed code is not possible (i.e. during a GC). In traditional
/// COM scenarios this is common, but scenarios involving <see href="https://docs.microsoft.com/windows/win32/api/windows.ui.xaml.hosting.referencetracker/nn-windows-ui-xaml-hosting-referencetracker-ireferencetrackertarget">Reference Tracker hosting</see>
/// calling of the IUnknown API during a GC is possible.
/// </remarks>
CallerDefinedIUnknown = 1,
/// <summary>
/// Flag used to indicate the COM interface should implement <see href="https://docs.microsoft.com/windows/win32/api/windows.ui.xaml.hosting.referencetracker/nn-windows-ui-xaml-hosting-referencetracker-ireferencetrackertarget">IReferenceTrackerTarget</see>.
/// When this flag is passed, the resulting COM interface will have an internal implementation of IUnknown
/// and as such none should be supplied by the caller.
/// </summary>
TrackerSupport = 2,
}
/// <summary>
/// Enumeration of flags for <see cref="ComWrappers.GetOrCreateObjectForComInstance(IntPtr, CreateObjectFlags, object?)"/>.
/// </summary>
[Flags]
public enum CreateObjectFlags
{
None = 0,
/// <summary>
/// Indicate if the supplied external COM object implements the <see href="https://docs.microsoft.com/windows/win32/api/windows.ui.xaml.hosting.referencetracker/nn-windows-ui-xaml-hosting-referencetracker-ireferencetracker">IReferenceTracker</see>.
/// </summary>
TrackerObject = 1,
/// <summary>
/// Ignore any internal caching and always create a unique instance.
/// </summary>
UniqueInstance = 2,
}
/// <summary>
/// Class for managing wrappers of COM IUnknown types.
/// </summary>
[CLSCompliant(false)]
public abstract partial class ComWrappers
{
/// <summary>
/// Interface type and pointer to targeted VTable.
/// </summary>
public struct ComInterfaceEntry
{
/// <summary>
/// Interface IID.
/// </summary>
public Guid IID;
/// <summary>
/// Memory must have the same lifetime as the memory returned from the call to <see cref="ComputeVtables(object, CreateComInterfaceFlags, out int)"/>.
/// </summary>
public IntPtr Vtable;
}
/// <summary>
/// ABI for function dispatch of a COM interface.
/// </summary>
public struct ComInterfaceDispatch
{
public IntPtr vftbl;
/// <summary>
/// Given a <see cref="System.IntPtr"/> from a generated VTable, convert to the target type.
/// </summary>
/// <typeparam name="T">Desired type.</typeparam>
/// <param name="dispatchPtr">Pointer supplied to VTable function entry.</param>
/// <returns>Instance of type associated with dispatched function call.</returns>
public static unsafe T GetInstance<T>(ComInterfaceDispatch* dispatchPtr) where T : class;
}
/// <summary>
/// Create an COM representation of the supplied object that can be passed to an non-managed environment.
/// </summary>
/// <param name="instance">A GC Handle to the managed object to expose outside the .NET runtime.</param>
/// <param name="flags">Flags used to configure the generated interface.</param>
/// <returns>The generated COM interface that can be passed outside the .NET runtime.</returns>
public IntPtr GetOrCreateComInterfaceForObject(object instance, CreateComInterfaceFlags flags);
/// <summary>
/// Compute the desired VTables for <paramref name="obj"/> respecting the values of <paramref name="flags"/>.
/// </summary>
/// <param name="obj">Target of the returned VTables.</param>
/// <param name="flags">Flags used to compute VTables.</param>
/// <param name="count">The number of elements contained in the returned memory.</param>
/// <returns><see cref="ComInterfaceEntry" /> pointer containing memory for all COM interface entries.</returns>
/// <remarks>
/// All memory returned from this function must either be unmanaged memory, pinned managed memory, or have been
/// allocated with the <see cref="System.Runtime.CompilerServices.RuntimeHelpers.AllocateTypeAssociatedMemory(Type, int)"/> API.
///
/// If the interface entries cannot be created and <code>null</code> is returned, the call to <see cref="ComWrappers.GetOrCreateComInterfaceForObject(object, CreateComInterfaceFlags)"/> will throw a <see cref="System.ArgumentNullException"/>.
/// </remarks>
protected unsafe abstract ComInterfaceEntry* ComputeVtables(object obj, CreateComInterfaceFlags flags, out int count);
/// <summary>
/// Get the currently registered managed object or creates a new managed object and registers it.
/// </summary>
/// <param name="externalComObject">Object to import for usage into the .NET runtime.</param>
/// <param name="flags">Flags used to describe the external object.</param>
/// <param name="wrapper">An optional <see cref="object"/> to be used as the wrapper for the external object</param>
/// <returns>Returns a managed object associated with the supplied external COM object.</returns>
/// <remarks>
/// Providing a <paramref name="wrapper"/> instance means <see cref="ComWrappers.GetOrCreateObjectForComInstance(IntPtr, CreateObjectFlags, object?)"/>
/// will not be called.
///
/// If the <paramref name="wrapper"/> instance already has an associated external object a <see cref="System.NotSupportedException"/> will be thrown.
/// </remarks>
public object GetOrCreateObjectForComInstance(IntPtr externalComObject, CreateObjectFlags flags, object? wrapper = null);
/// <summary>
/// Create a managed object for the object pointed at by <paramref name="externalComObject"/> respecting the values of <paramref name="flags"/>.
/// </summary>
/// <param name="externalComObject">Object to import for usage into the .NET runtime.</param>
/// <param name="flags">Flags used to describe the external object.</param>
/// <returns>Returns a managed object associated with the supplied external COM object.</returns>
/// <remarks>
/// If the object cannot be created and <code>null</code> is returned, the call to <see cref="ComWrappers.GetOrCreateObjectForComInstance(IntPtr, CreateObjectFlags, object?)"/> will throw a <see cref="System.ArgumentNullException"/>.
/// </remarks>
protected abstract object? CreateObject(IntPtr externalComObject, CreateObjectFlags flags);
/// <summary>
/// Called when a request is made for a collection of objects to be released.
/// </summary>
/// <param name="objects">Collection of objects to release.</param>
/// <remarks>
/// The default implementation of this function throws <see cref="System.NotImplementedException"/>.
/// </remarks>
protected virtual void ReleaseObjects(IEnumerable objects);
/// <summary>
/// Register this class's implementation to be used as the single global instance.
/// </summary>
/// <remarks>
/// This function can only be called a single time. Subsequent calls to this function will result
/// in a <see cref="System.InvalidOperationException"/> being thrown.
///
/// Scenarios where the global instance may be used are:
/// * Object tracking via the <see cref="CreateComInterfaceFlags.TrackerSupport" /> and <see cref="CreateObjectFlags.TrackerObject" /> flags.
/// * Usage of COM related Marshal APIs.
/// </remarks>
public void RegisterAsGlobalInstance();
/// <summary>
/// Get the runtime provided IUnknown implementation.
/// </summary>
/// <param name="fpQueryInterface">Function pointer to QueryInterface.</param>
/// <param name="fpAddRef">Function pointer to AddRef.</param>
/// <param name="fpRelease">Function pointer to Release.</param>
protected static void GetIUnknownImpl(out IntPtr fpQueryInterface, out IntPtr fpAddRef, out IntPtr fpRelease);
}
}Example usage
The below example is merely for illustrative purposes. In a production ready consumption of the API many of the Marshal APIs would not be used and the VTable layouts should be done in a static manner for optimal efficiency.
[Guid("197BC142-7A71-4637-B504-894DE79C4A22")]
interface IPrint
{
public void PrintInt(int i);
}
class Print : IPrint
{
public void PrintInt(int i)
{
Console.WriteLine($"{nameof(IPrint.PrintInt)} - 0x{i:x}");
}
}
struct IUnknownVftbl
{
public IntPtr QueryInterface;
public IntPtr AddRef;
public IntPtr Release;
}
struct IPrintVftbl
{
public IUnknownVftbl IUnknownImpl;
public IntPtr PrintInt;
public delegate int _PrintInt(IntPtr thisPtr, int i);
public static _PrintInt pPrintInt = new _PrintInt(PrintIntInternal);
public static int PrintIntInternal(IntPtr dispatchPtr, int i)
{
unsafe
{
try
{
ComWrappers.ComInterfaceDispatch.GetInstance<IPrint>((ComWrappers.ComInterfaceDispatch*)dispatchPtr).PrintInt(i);
}
catch (Exception e)
{
return e.HResult;
}
}
return 0; // S_OK;
}
}
struct VtblPtr
{
public IntPtr Vtbl;
}
class IExternalObject
{
private struct IExternalObjectVftbl
{
public IntPtr QueryInterface;
public _AddRef AddRef;
public _Release Release;
public _AddObjectRef AddObjectRef;
public _RemoveObjectRef DropObjectRef;
}
private delegate int _AddRef(IntPtr This);
private delegate int _Release(IntPtr This);
private delegate int _AddObjectRef(IntPtr This, IntPtr o);
private delegate int _RemoveObjectRef(IntPtr This, IntPtr o);
private readonly IntPtr instance;
private readonly IExternalObjectVftbl vtable;
public IExternalObject(IntPtr instance)
{
var inst = Marshal.PtrToStructure<VtblPtr>(instance);
this.vtable = Marshal.PtrToStructure<IExternalObjectVftbl>(inst.Vtbl);
this.instance = instance;
}
~IExternalObject()
{
if (this.instance != IntPtr.Zero)
{
this.vtable.Release(this.instance);
}
}
public void AddObjectRef(object inst)
{
...
}
public void DropObjectRef(object inst)
{
...
}
}
class MyComWrappers : ComWrappers
{
protected unsafe override ComInterfaceEntry* ComputeVtables(object obj, CreateComInterfaceFlags flags, out int count)
{
IntPtr fpQueryInteface = default;
IntPtr fpAddRef = default;
IntPtr fpRelease = default;
ComWrappers.GetIUnknownImpl(out fpQueryInteface, out fpAddRef, out fpRelease);
var tables = new List<ComInterfaceEntry>();
var vtbl1 = new IPrintVftbl()
{
IUnknownImpl = new IUnknownVftbl()
{
QueryInterface = fpQueryInteface,
AddRef = fpAddRef,
Release = fpRelease
},
PrintInt = Marshal.GetFunctionPointerForDelegate(IPrintVftbl.pPrintInt)
};
var vtblRaw1 = RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(IPrintVftbl), sizeof(IPrintVftbl));
Marshal.StructureToPtr(vtbl1, vtblRaw1, false);
tables.Add(new ComInterfaceEntry { IID = IID_IPrint, Vtable = vtblRaw1 });
if (flags.HasFlag(CreateComInterfaceFlags.CallerDefinedIUnknown))
{
var vtbl2 = new IUnknownVftbl()
{
QueryInterface = fpQueryInteface,
AddRef = fpAddRef,
Release = fpRelease
};
var vtblRaw2 = RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(IUnknownVftbl), sizeof(IUnknownVftbl));
Marshal.StructureToPtr(vtbl2, vtblRaw2, false);
tables.Add(new ComInterfaceEntry { IID = IID_IUnknown, Vtable = vtblRaw2 });
}
// Return pointer to memory containing ComInterfaceEntry collection
}
protected override object CreateObject(IntPtr externalComObject, CreateObjectFlags flags)
{
return new IExternalObject(externalComObject);
}
}/cc @jkotas @Scottj1s @dunhor @jkoritzinsky @davidwrighton @terrajobst @tannergooding @jeffschwMSFT
Activity
jkoritzinsky commentedon Jan 17, 2020
Would it be possible to make the return value of
ComputeVtablespinned by the caller instead of by the implementation? That would remove a "pit of failure" around using a managed array as storage for the vtables and using the implicit conversion toReadOnlySpan<T>.jkotas commentedon Jan 17, 2020
There is no way to pin byrefs via GCHandles (it would be expensive for the GC to allow it). If you believe that
ReadOnlySpan<T>in the signature is a pit of failure, it may be better to fix it by makingComputeVtablesreturn unmanaged pointer + count explicitly.jkotas commentedon Jan 17, 2020
This assumes that the vtables won't be ever unloaded. I think it is ok for v1, but I wondering whether we need to do any tweaks in the design now to potentially allow unloading these in future.
AaronRobinsonMSFT commentedon Jan 21, 2020
That is just an example. I hope the preface for the Example section wasn't missed. That example is for conceptual understanding for the API review. It is not how I would expect source generators to consume the API.
I don't think so. The source generator here can drive that decision. Once all the CCWs the source generated are unused, that memory can be reclaimed. I would argue that is something the source generator code can track themselves.
AaronRobinsonMSFT commentedon Jan 21, 2020
Completely forgot about the implicit conversion. That seems bad actually. I think I am inclined to change that to one of the original designs and return a tuple with a pointer and count - similar to what @jkotas is suggesting above.
jkotas commentedon Jan 21, 2020
What is the code that the source generator would generate to track this?
AaronRobinsonMSFT commentedon Jan 21, 2020
Boo. I just re-realized the whole point of this is for WinUI and in that scenario the caller isn't going to be implementing any part of
IUnknown. I was originally going to say the CCW could wrap theRelease()call and when that returns 0 for all CCWs with the associated memory it could be released. However, that won't work for cases when the source generator code can't provide anyRelease()implementation due to the aforementioned GC issues.This is something we should put thought into now. I am loathe to request a callback for when an object is destroyed. Perhaps some kind of enumerating mechanism for the CCW caches? Open to other suggestions as well.
AaronRobinsonMSFT commentedon Jan 21, 2020
Thinking more about this, I believe I got distracted from my example and @jkotas's very specific question.
@jkotas Users should be able to track this information by returning their own version of
Release()that proxies to the default one supplied. Of course this will not work in WinUI scenarios. Perhaps that makes it worth the effort to try and think of a solution, but in general I hope not for the following reason.I hope that no one will use
Marshal.AllocCoTaskMem(). Instead I expect users would pin some static class memory and provide the address to the array in the callback. This would allow the memory to be coupled to the assembly loader which will live as long as the object stays around and the object itself will be extended to as long as the CCW.This means that when the CCW is freed, the managed object can be collected, which means the assembly and all associated memory can be collected.
Is that sound or am I missing something?
jkotas commentedon Jan 21, 2020
How does this coupling happen? Does this mean that all structures involved in this have to be allocated on GC heap?
Also, note that infinite pinning is not good for GC performance - it prevents GC from compacting the heap and getting rid of the fragmentation.
A different way to solve this would be to introduce method that allocates unmanaged piece of memory that has lifetime attached to lifetime of given Type. E.g.
IntPtr AllocateMemoryAssociatedWithType(Type type, int size). Implementing this would be cheap - we do have a facility like this internally in the runtime (LoaderAllocator). All calls of AllocCoTaskMem in your example would be replaced with this method.AaronRobinsonMSFT commentedon Jan 21, 2020
Since the data would be the same for all instances of a type, I assume it would something along the lines of the definition below. Then the
LongLivedDatawould be associated with the assembly that it lives in. The trick is ensuring the underlying data forType1never moves. My assumption here was pinned memory or some other mechanism I don't know about but was sure the runtime had could be used to efficiently make that memory position stable/static.If there isn't any such efficient mechanism then your allocator API seems reasonable.
1 remaining item
mjsabby commentedon Feb 11, 2020
Will we support this xplat? That would be awesome.
AaronRobinsonMSFT commentedon Feb 11, 2020
@mjsabby That is a good question. For right now the implementation is being written to target only Windows.
However, the implementation has a single dependency on a Windows API and that is only because of COM apartments. On non-Windows platforms there isn't any concept of COM apartments so that could easily be severed with little fanfare.My plan is to ensure the API exists on Windows for future WinUI/WinRT scenario tooling. If it proves to be valuable outside of that scenario it would be a small amount of work to enable this xplat if demand exists.
Edit: The COM Apartment API usage has been removed.
mjsabby commentedon Feb 11, 2020
@AaronRobinsonMSFT Right, if we can cull that it would be a very useful addition to cross platform COM porting. I've seen many libraries benefit from the conventions of COM that don't need registry based activation or COM and would be a breeze to support with the right tooling support, and I say that even if the tooling continues to be windows-based.
Consider a +1 from me :)
terrajobst commentedon Mar 3, 2020
Video
Vtablevs.vftblAPI
jkoritzinsky commentedon Mar 3, 2020
Design has been validated by the CsWinRT team.
hughbe commentedon Apr 19, 2020
@AaronRobinsonMSFT Would this API allow for converting an IntPtr into a managed ComImport interface? I'm thinking about the following use case as an example
The class
ControlimplementsIPersistso obviously its com callable wrapper can query theIPersistinterface. This means I can passcontrolto interop/PInvoke where the native code for example expects anIPersistobject.However, what I actually want to do is to call methods on the
IPersistinterface and call methods against it from c# just as I could do from native code.I can't directly cast
controlto the interface here, as the interface thatControlimplements is internal to its own assembly, but is exposed to COM.Any ideas?
AaronRobinsonMSFT commentedon Apr 19, 2020
@hughbe I am going to assume we are talking about the WinForms
Control.This API could perform the desired actions, but that would be a decent amount of work and isn't scalable. VTable layouts would need to be defined and allocated and most of that code is monotonous and easy to get wrong. There is a test that shows an example of consuming a trivial
IUnknownbased interface.Instead I would recommend using the
dynamickeyword and simply call the method. The COM support fordynamicuses reflection and doesn't respect access modifiers so you should be able to call the desired function as follows:If the
dynamickeyword is too much magic, it is possible to use reflection manually and call the desired function, but that code is very verbose.The
ComWrappersAPI is intended for a much lower level scenario - the entire replacement of the built-in CCW/RCW support. This API coupled with a source generation process can be used to avoid any code generation at run time and thus be more AOT friendly. Additionally it permits consumers to define marshaling semantics instead of relying on runtime changes. This API is intended for building tools like SharpGenTools.