Difference between revisions of "Data Storage"
(→Proper use of Filesystems) |
|||
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | Here we outline some best practices for data storage. | + | Here we outline some best practices for data storage on the DeepSense platform. |
== Proper use of Filesystems == | == Proper use of Filesystems == | ||
| Line 12: | Line 12: | ||
== Best Practices == | == Best Practices == | ||
| + | * Check your space usage regularly to make sure you are below your quota | ||
* Small amounts of data may be transferred via the login nodes, using ''scp'' or ''rsync'' | * Small amounts of data may be transferred via the login nodes, using ''scp'' or ''rsync'' | ||
* Medium amounts of data '''must''' be transferred using the protocol nodes | * Medium amounts of data '''must''' be transferred using the protocol nodes | ||
| Line 17: | Line 18: | ||
* Regularly remove temporary data stored in the scratch filesystem, as this will be used for vast amounts of temporary data | * Regularly remove temporary data stored in the scratch filesystem, as this will be used for vast amounts of temporary data | ||
* Data no longer used should be removed, as DeepSense is not intended for long term storage | * Data no longer used should be removed, as DeepSense is not intended for long term storage | ||
| − | |||
* When your project/account are closed, please remove all data | * When your project/account are closed, please remove all data | ||
* To help us avoid restricting the storage policies, please use the space conscientiously | * To help us avoid restricting the storage policies, please use the space conscientiously | ||
Latest revision as of 15:50, 18 December 2020
Here we outline some best practices for data storage on the DeepSense platform.
Proper use of Filesystems
Storing your data in the proper location is important as it will help everyone on your project locate and access the data.
Your home directory should be used for your personal scripts, or test code. Since only you have access to your home directory, it is not a good place to store your data. Instead, store your data in the data filesystem where everyone in your project has access to it.
The scratch filesystem is meant for temporary data only. It is not backed up. Again, anyone in your project group will have access to this filesystem.
Best Practices
- Check your space usage regularly to make sure you are below your quota
- Small amounts of data may be transferred via the login nodes, using scp or rsync
- Medium amounts of data must be transferred using the protocol nodes
- Large amounts of data can be transferred via the protocol nodes, using samba. Alternatively, you can bring an external hard drive to us and we can directly transfer your data
- Regularly remove temporary data stored in the scratch filesystem, as this will be used for vast amounts of temporary data
- Data no longer used should be removed, as DeepSense is not intended for long term storage
- When your project/account are closed, please remove all data
- To help us avoid restricting the storage policies, please use the space conscientiously
