I work on an HPC and often I have to share files with other users. The most approachable solution is to have an external cloud storage and recline back and forth. However there’s some projects that are quite heavy (several TB) and that is unfeasible. We do not have a shared group. The following is the only solution I found which is not to just set al permissions to 777, and I still don’t like it.
Create a directory and set ACL to give access to the selected users. This works fine if the users create new files in there, but it does not work if they copy from somewhere else as default umask is 022. Thus the only appropriate solution is to change default umask to 002, which however affects file creation system wide. The alternative is to change permissions every time you copy something, but you all know very well that is not going to happen.
Does it really have to be such a pain in the ass?
Lots of paperwork involved around our HPC. Politica in the middle and plausibly would have to sign confidentiality agreement with everyone who shares anything in there, which would have to go through a review process which is generally takes about 6 months.
There’s a reason for this governance and you’re putting your whole team at risk trying to do this yourself
It’s just default Linux permissions. People who get access to the HPC through the same institution are placed in the same group. As you may know scientific collaboration is quite important. Indeed when collaborating we sign all the necessary paperwork, but that does not translate to the HPC administrators who are part of a separate institution. To request a separate group you have to contact the HPC institution, they will have to contact the institutions involved. Those institutions will have to check that NDAs are already in place. If the NDAs are in place they will have to check that the data to be shared is actually covered by the project. I will have to fill a bunch of paperwork. This will be sent to an external auditor to check that everything is correct, and then everything goes back in that chain.
I already waste way too much of my time on paperwork. Worst case scenario is a collaborator leaks some data which will be published publicly in a few months anyway. And those collaborators will have access to such data anyway, just through other less comfortable means.