Skip to content

Can't stop circular dataflow blocks #612

Description

@ZhaoXiangXML

I'm trying to implement a crawler with System.Threading.Tasks.Dataflow, the core feature is to have a TransformBlock, which accepts an url as input, request the url, parse the response, and find more urls for the same block.

Then I noticed a problem that I cannot complete/stop this pipeline. The following code simulates my problem:

using System;
using System.Threading;
using System.Threading.Tasks.Dataflow;

namespace ConsoleApp1
{
    class Program
    {
        static void Main()
        {
            int count = 0;

            var block = new TransformBlock<string, string>(uri =>
            {
                //await Task.Delay(10);
                Console.WriteLine($"Downloading {count}");

                if (Interlocked.Increment(ref count) > 10)
                {
                    return null;
                }

                return uri;
            });

            var linkOptions = new DataflowLinkOptions { PropagateCompletion = false };

            block.LinkTo(block, linkOptions, (x) => !string.IsNullOrEmpty(x));

            block.Post("");
            block.Post("");

            while (true)
            {
                Console.WriteLine($"Check Count: {count}");

                if (count >= 2)
                {
                    Console.WriteLine("Complete");
                    block.Complete();
                    break;
                }

                Thread.Sleep(20);
            }

            Console.WriteLine("After Check");

            block.Completion.Wait();

            Console.WriteLine("Done");
        }
    }
}

The output is:

Check Count: 0
Downloading 0
Downloading 1
Check Count: 2
Complete
After Check

And program stuck here, block.Completion.Wait() never return.

I've also tried to have two blocks link to each other, yet the result is same.

So what can I do about my code? How can I stop this dataflow?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions