Repository navigation
Document the usage of temp rather than $HOME for keeping temporary data files #218
Description
Message from Yuvi on Slack:
... The issue here is that writing data to $HOME is very slow, and it is also shared across all users. The cell that got stuck was trying to write ~1GB of data to $HOME, and when spread across the 100+ users, it turned everything super slow! This is one of the reasons 'cloud native' workflows directly doing object storage are faster, because they don't have to touch possibly slow local disks. $HOME is designed to store code, rather than data.
The solution here is to use the temporary directory to keep temporary data files. These will reset each time the user server restarts, and are also much faster. Plus they are not shared across all users. This also works across local machines and any cloud providers. The python tempfile standard library module is probably very helpful here.
So the upshot here is don't use $HOME to store data. It also means it doesn't get cleaned up, and will cost money sort of indefinitely into the future as well. Plus, it leads to issues when doing workshops. Use tempfile if you need to download data locally.
I hope this was helpful! I think it'll also be helpful for this to be set up in some sort of outside documentation, but not sure where.
Metadata
Metadata
Assignees
Labels
Type
Fields
Area of Work
Activity
yuvipanda commentedon Mar 22, 2024
An additional point here is that while $HOME usually has a lot of space,
/tmpdoes not. As an example, if I run the command that prints out disk usage (df -h) in a terminal on the openscapes hub, I get:There's a bunch of extra stuff here (including stuff that says
tmpfsbut you can ignore that), but primary use to us is two:We can tell that the temporary directory is under
/because of the following python code:And looking at the list of various mountpoints in the
df -houtput, you can see that/is the one that/tmpis under. If there was a specific mount for/tmpit would show up there!Anyway, the thing to note here is that there is 57G available for all users on a single node when using
/tmp. You can't see other people's /tmp, but it does take up space! And which two users are placed on a node is not upto the users, so it is possible for a user to use up all the space available for/tmpin a particular node.yuvipanda commentedon Mar 22, 2024
I've discussed some potential solutions in 2i2c-org/infrastructure#3833, but don't believe we should put more effort into them at this point.