Data Storage
Here we outline some best practices for data storage on the DeepSense platform.
Proper use of Filesystems
Storing your data in the proper location is important as it will help everyone on your project locate and access the data.
Your home directory should be used for your personal scripts, or test code. Since only you have access to your home directory, it is not a good place to store your data. Instead, store your data in the data filesystem where everyone in your project has access to it.
The scratch filesystem is meant for temporary data only. It is not backed up. Again, anyone in your project group will have access to this filesystem.
Best Practices
- Check your space usage regularly to make sure you are below your quota
- Small amounts of data may be transferred via the login nodes, using scp or rsync
- Medium amounts of data must be transferred using the protocol nodes
- Large amounts of data can be transferred via the protocol nodes, using samba. Alternatively, you can bring an external hard drive to us and we can directly transfer your data
- Regularly remove temporary data stored in the scratch filesystem, as this will be used for vast amounts of temporary data
- Data no longer used should be removed, as DeepSense is not intended for long term storage
- When your project/account are closed, please remove all data
- To help us avoid restricting the storage policies, please use the space conscientiously
