Skip to content

Is there any way to limit GPU memory? #102

Closed
Closed
Is there any way to limit GPU memory?#102

Description

@seungtaek94
seungtaek94
opened on Mar 9, 2021

Hi.

I know that tensorflow, allocate almost gpu memory when it initialize because of the memory fragmentation.

But other process that in my applications, also need to use GPU. So i need to limit the tensorflow gpu usages.

In python can handle this problem like below:

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=2048)])
  except RuntimeError as e:

Is there any way to limit GPU memory in cppflow or tensorflow c_api ?

Activity

  1. serizba commented on Mar 9, 2021

    @serizba
    serizba
    on Mar 9, 2021
    Owner

    Hi @seungtaek94 ,

    Yes, you can specify the context options. First you need to obtain them in a serialized way, check this to see how.

    Then you can set the global context with a new one with your defined options.

    const std::vector<uint8_t> serialized_options = { 0x32, 0xb, 0x9, 0x9a, 0x99, 0x99, 0x99, 0x99, 0x99, 0xb9, 0x3f, 0x20, 0x1 };
TFE_ContextOptions* options = TFE_NewContextOptions();
TFE_ContextOptionsSetConfig(options, static_cast<const void*>(serialized_options.data()), serialized_options.size(), cppflow::context::get_status());
cppflow::get_global_context() = cppflow::context(options);

    Hope it helps!

  2. seungtaek94 commented on Mar 11, 2021

    @seungtaek94
    seungtaek94
    on Mar 11, 2021
    ContributorAuthor

    @serizba

    Thanks for ur helps :)

    But is there no way to use only c++ without python code?.

    And that code generate all same serialized_options like below :(

    Create serialized options which allow TF to use a certain percentage of GPU memory and allow TF to expand this memory if required.
GPU memory to be used: 10.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 20.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 30.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 40.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 50.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 60.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 70.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 80.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used: 90.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}

  3. ljn917 commented on Mar 11, 2021

    @ljn917
    ljn917
    on Mar 11, 2021 via email
    Contributor

  4. seungtaek94 commented on Mar 11, 2021

    @seungtaek94
    seungtaek94
    on Mar 11, 2021
    ContributorAuthor

    @ljn917 , I just tried ur comment function TF_CreateConfig like below:

    std::unique_ptr<TF_Buffer, decltype(&TF_DeleteBuffer)> config(
		TF_CreateConfig(
			 false, /*xla*/
			 false, /* gpu_memory_allow_growth */
			1 /* num_cpu_devices */
			),
		TF_DeleteBuffer);

	TFE_ContextOptions* options = TFE_NewContextOptions();
	TFE_ContextOptionsSetConfig(options, config->data, config->length, cppflow::context::get_status());
	cppflow::get_global_context() = cppflow::context(options);

    but it doesnt help for me..

    if i set gpu_memory_allow_growth = false, it has took 7386MiB/8192MiB memory in rtx 2080 super.
    otherwise if i set true, it has took 7002MiB / 8192MiB. Still takes to much..

    My model has only 3M number of params. hmm..

    Can you try TF_CreateConfig function? Btw, it is currently experimental. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/c/c_api_experimental.h#L93
    …
    On Thu, Mar 11, 2021, 01:40 Seungtaek Kim @.***> wrote: @serizba https://github.com/serizba Thanks for ur helps :) But is there no way to use only c++ without python code?. And that code generate all same serialized_options like below :( Create serialized options which allow TF to use a certain percentage of GPU memory and allow TF to expand this memory if required. GPU memory to be used: 10.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 20.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 30.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 40.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 50.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 60.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 70.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 80.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} GPU memory to be used: 90.0 %, {0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1} — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALDTHXU47NP4UTISOOU3H3TDBQU3ANCNFSM4Y2OL37A .

  5. serizba commented on Mar 11, 2021

    @serizba
    serizba
    on Mar 11, 2021
    Owner

    And how much GPU memory does it take when you run it on python?

  6. seungtaek94 commented on Mar 11, 2021

    @seungtaek94
    seungtaek94
    on Mar 11, 2021
    ContributorAuthor

    @serizba

    • without any gpu options: 7405MiB / 8192MiB
    • with set_memory_growth: 7024MiB / 8192MiB
    • with tf.config.experimental.VirtualDeviceConfiguration:
      (memory_limit=2048): 3273MiB / 8192MiB
      (memory_limit=1024): 2254MiB / 8192MiB
      (memory_limit=512): 1744MiB / 8192MiB
      (memory_limit=256): OOM

    And how much GPU memory does it take when you run it on python?

  7. ljn917 commented on Mar 11, 2021

    @ljn917
    ljn917
    on Mar 11, 2021
    Contributor

    @seungtaek94 Can you test #105?

  8. seungtaek94 commented on Mar 11, 2021

    @seungtaek94
    seungtaek94
    on Mar 11, 2021
    ContributorAuthor

    @ljn917

    i tested that code, and it works. And now it shows different serialized result like that sample :)

    i have one more question about this serialized options.

    this serialized results r same on every system or tf version?.

    i serialized with tf 2.4 with rtx 2080 super, and that result is same with ur samples in that commit #105.

    thanks for ur help :)

    Create serialized options which allow TF to use a certain percentage of GPU memory and allow TF to expand this memory if required.
GPU memory to be used:  10.0 %
{0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xb9,0x3f,0x20,0x1}
GPU memory to be used:  20.0 %
{0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xc9,0x3f,0x20,0x1}
GPU memory to be used:  30.000000000000004 %
{0x32,0xb,0x9,0x34,0x33,0x33,0x33,0x33,0x33,0xd3,0x3f,0x20,0x1}
GPU memory to be used:  40.0 %
{0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xd9,0x3f,0x20,0x1}
GPU memory to be used:  50.0 %
{0x32,0xb,0x9,0x0,0x0,0x0,0x0,0x0,0x0,0xe0,0x3f,0x20,0x1}
GPU memory to be used:  60.00000000000001 %
{0x32,0xb,0x9,0x34,0x33,0x33,0x33,0x33,0x33,0xe3,0x3f,0x20,0x1}
GPU memory to be used:  70.0 %
{0x32,0xb,0x9,0x67,0x66,0x66,0x66,0x66,0x66,0xe6,0x3f,0x20,0x1}
GPU memory to be used:  80.0 %
{0x32,0xb,0x9,0x9a,0x99,0x99,0x99,0x99,0x99,0xe9,0x3f,0x20,0x1}
GPU memory to be used:  90.0 %
{0x32,0xb,0x9,0xcd,0xcc,0xcc,0xcc,0xcc,0xcc,0xec,0x3f,0x20,0x1}

    @seungtaek94 Can you test #105?

  9. ljn917 commented on Mar 11, 2021

    @ljn917
    ljn917
    on Mar 11, 2021
    Contributor

    The protocol buffer format should be platform independent. The only possibility of incompatibility is that the protocol buffer library is updated to a non-backward compatible version, which is very unlikely.

  10. seungtaek94 commented on Mar 11, 2021

    @seungtaek94
    seungtaek94
    on Mar 11, 2021
    ContributorAuthor

    thanks for all answer :)

  11. seungtaek94
    closed this as completedon Mar 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions