Skip to content

InvalidCastException thrown after TimerQueueTimer.CallCallback #81211

Closed
Closed
InvalidCastException thrown after TimerQueueTimer.CallCallback#81211
Assignees
lambdageek
Labels
area-VM-threading-monoos-iosApple iOS
Milestone
7.0.x

Description

@tipa
tipa
opened on Jan 26, 2023

Description

After upgrading my iOS app from classic Xamarin.iOS to .NET7 I started observing frequent InvalidCastExeptions in my crash reporting tool (AppCenter). I haven't changed much dependency-wise and I am unable to reproduce this locally.

Reproduction Steps

Unfortunately, I don't know what part of my code causes this behavior.
For the first stack trace, I assume that my code (or some dependencies code) is creating a CancellationTokenSource with a delay parameter.
For the second stack trace below, I assume that some call to Task.Delay is causing the crash.

Expected behavior

No crash :)

Actual behavior

SIGABRT: Arg_InvalidCastException 

System.Threading.CancellationTokenSource.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs

Line 35 in 215b39a

((CancellationTokenSource)state!).NotifyCancellation(throwOnFirstException: false); // skip ThrowIfDisposed() check in Cancel()

SIGABRT: Arg_InvalidCastException 

System.Threading.Tasks.Task.DelayPromise.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs

Line 5632 in 26b58b9

private static void TimerCallback(object? state) => ((DelayPromise)state!).CompleteTimedOut();

Both crashes originate from here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs

Line 706 in 26b58b9

_timerCallback(_state);

The "_state" variable is nullable, and based on the reports, the crashes are caused by the variable to be indeed null.

Regression?

I did not see those crashes when using "old" Xamarin.iOS

Known Workarounds

No response

Configuration

.net7-ios
happens on various iPhone models and iOS versions

Other information

No response

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Jan 26, 2023
  2. ghost added
    area-System.Threading
    on Jan 26, 2023
  4. stephentoub
    added
    area-VM-threading-mono
    os-iosApple iOS
    on Jan 27, 2023
  6. stephentoub
    removed
    area-System.Threading
    on Jan 27, 2023

  7. akoeplinger commented on Jan 27, 2023

    @akoeplinger
    akoeplinger
    on Jan 27, 2023
    Member

    /cc @lambdageek

  8. lambdageek commented on Jan 27, 2023

    @lambdageek
    lambdageek
    on Jan 27, 2023
    Member

    Weird. Both CancellationTokenSource and DelayTimer pass this for the state argument of TimerQueueTimer.
    So the state passed to the callback should be non-NULL, and the cast from object to the respective types should succeed.

    Maybe there's a GC hole somewhere?

  9. lambdageek commented on Jan 27, 2023

    @lambdageek
    lambdageek
    on Jan 27, 2023
    Member

    @tipa when you try to reproduce the issue locally, are you trying it in a simulator or on a device?

  10. tipa commented on Jan 27, 2023

    @tipa
    tipa
    on Jan 27, 2023
    Author

    @lambdageek I tested my apps on both simulator and physical device numerous times - I was never able to observe this myself (edit: I now am, using a TestFlight app and repeatedly starting it).
    The crash is not super-common, but also not super-rare. I updated both my apps ~1 week ago.
    Since then, one of the two apps that I am observing this is installed on ~5000 devices with around 2000 app sessions per day and the crash occurred 7 times (on 7 different devices).
    And the other app is installed on ~1000 devices with around 300 app sessions per day and the crash occurred 3 times (on 2 different devices).

  11. tipa commented on Jan 31, 2023

    @tipa
    tipa
    on Jan 31, 2023
    Author

    I am also seeing a lot of these exceptions (220 crashes from 80 devices , all originating from DelayPromise) in an app that does a lot of (concurrent) network calls. Maybe the HttpClient uses this under the hood...

  12. tipa commented on Feb 15, 2023

    @tipa
    tipa
    on Feb 15, 2023
    Author

    A lot more crash reports have been accumulated in the meantime. I expect the real number to be even higher because AppCenter only seems to report a fraction of the real crashes.
    I was also able to reproduce this now in a TestFlight app. it appears to happen when the app is of spawning a few (~10) new tasks and letting them do some concurrent work (getting credentials from the key chain, doing HTTP requests, deserialize JSON (using System.Text.Json), etc).
    The other app I mentioned above (now 322 crashes in the last 30 days) does something similar.

    I observed a crash with a similar behavior on Android (see here + fix here), maybe this is from a similar cause?

    Additionally, I also started seeing these stack traces::

    SIGABRT: Object reference not set to an instance of an object 

    System.AggregateException..ctor(String , List`1 )
System.Threading.Tasks.TaskExceptionHolder.Finalize()

    SIGABRT: Arg_ArrayTypeMismatchException 

    System.AggregateException..ctor(String , List`1 )
System.Threading.Tasks.TaskExceptionHolder.Finalize()

    Maybe I should have opened this bug in the xamarin-macios repo?

  13. steveisok commented on Feb 15, 2023

    @steveisok
    steveisok
    on Feb 15, 2023
    Member

    Maybe I should have opened this bug in the xamarin-macios repo?

    No, it does look like a runtime issue, so you have the right place.

  14. steveisok
    assigned
    lambdageek
    on Feb 15, 2023

  15. 29 remaining items

  16. lateralusX commented on Oct 4, 2023

    @lateralusX
    lateralusX
    on Oct 4, 2023
    Member

    Yes, if you have got passed method_init and still sees NULL values in used GOT slots for that method, that probably means you hit the race where that thread exited its call to method_init before the stores into needed GOT slot has happened or became visible by the other thread. This is an LLVM only issue, meaning that it won't reproduce when running without LLVM. When I originally investigate that issue it caused rare random crashes that could occur at any point during apps lifetime, because a method calls its method_init only on first call to to the method, and it needs to race with another thread doing the same thing to expose the potential race.

    This fix has been implemented and used in a downstream repo running in some large apps, installed and executed in very large quantities for over a year, and it eliminated the in-frequent crashes previously seen by those apps and didn't cause any regression (x64). It has been in dotnet/runtime main for over a year, so since the fix have been around for sometime and it has been applied to both mono/mono and dotnet/runtime repro's. I believe it should be a small risk of backporting it to net7, especially since we see issues potentially affected by it and its LLVM only.

  17. divil5000 commented on Oct 4, 2023

    @divil5000
    divil5000
    on Oct 4, 2023

    We have never seen this issue in the past, when running on the Xamarin framework. Only since our latest release where we changed to net7-ios. In both cases we are using LLVM. Is that expected? It seems to contradict what @lateralusX is saying.

  18. michaldobrodenka commented on Oct 4, 2023

    @michaldobrodenka
    michaldobrodenka
    on Oct 4, 2023

    We have never seen this issue in the past, when running on the Xamarin framework. Only since our latest release where we changed to net7-ios. In both cases we are using LLVM. Is that expected? It seems to contradict what @lateralusX is saying.

    I haven't seen 1 crash of this type after we turned off LLVM. We had crash like this about every day.

  19. lateralusX commented on Oct 4, 2023

    @lateralusX
    lateralusX
    on Oct 4, 2023
    Member

    We have never seen this issue in the past, when running on the Xamarin framework. Only since our latest release where we changed to net7-ios. In both cases we are using LLVM. Is that expected? It seems to contradict what @lateralusX is saying.

    Xamarin runs on a Mono branch 2020-2 and the critical changes around this race was introduced later in both mono/mono as well as dotnet/runtime, so running on a later mono/mono branch as well as dotnet/runtime branch will hit it, but not the branch used by legacy Xamarin. That might explain what you have been identifying. On the product I originally hit and analyzed the issue we didn't see it until after upgrading to a later mono/mono commit, not included in Mono's 2020-2 branch.

  20. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Oct 4, 2023
  21. lambdageek
    modified the milestones: 8.0.0, 7.0.x on Oct 4, 2023

  22. lambdageek commented on Oct 4, 2023

    @lambdageek
    lambdageek
    on Oct 4, 2023
    Member

    Thanks for the detailed explanation (and I missed that we changed this code relatively recently - that explains why Xamarin wasn't affected).

    I'm going to try a local build of #93006 to verify that it makes the crash go away (It's quite infrequent for me under Xcode so if I don't see if after a couple hundred launches, I'm going to consider it resolved).

  23. lambdageek
    added a commit that references this issue on Oct 6, 2023

    [release/7.0] [Mono] Race in init_method when using LLVM AOT. (#93006)

    7ee35b9
  24. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Oct 6, 2023

  25. divil5000 commented on Oct 24, 2023

    @divil5000
    divil5000
    on Oct 24, 2023

    So where are we with this issue? Will we have to move to net8-ios to fix it (when it's released) or will there be a patch to net7-ios at some point?

  26. lambdageek commented on Oct 24, 2023

    @lambdageek
    lambdageek
    on Oct 24, 2023
    Member

    It's fixed in .NET 8 RC2, and there is a backport that will be in an upcoming .NET 7 servicing release

  27. tipa commented on Nov 4, 2023

    @tipa
    tipa
    on Nov 4, 2023
    Author

    @lambdageek thanks a lot for the fix - the crashes I reported in my first post of this thread have gone away!
    However I still see those Exceptions (both the ArrayTypeMismatchException & NullReferenceExceptions) that I reported in this comment. Should I file a separate bug report for them? They also started to happen after migrating from Xamarin.iOS to .NET7

  28. lewing
    closed this as completedon Feb 9, 2024
  29. github-actions
    locked and limited conversation to collaborators on Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-VM-threading-monoos-iosApple iOS

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions