Wrangler Maintenance 13 March 2018

Posted by Mitchell Collins-Bailey on Mar 9, 2018 5:07:28 PM

Updated on Mar 27, 2018 4:41:16 PM

After extensive hardware issues and then filesystem recovery attempts Wrangler now has /gpfs/flash available cluster-wide. We've ended up with a number of files in the lost+found location after the fsck repair programs finally completed cleanly. /gpfs/flash is the scratch filesystem in the Wrangler cluster so we expect files to be replicated elsewhere but if there were any files that are of very high value that are missing there's a chance that they can be restored. Please submit a consulting ticket and the Wrangler admins will determine if they can be restored. Please note that the lost+found recovery location is the combination of all files in no particular order or structure and requires substantial sorting to track down individual entries.

Updated on Mar 13, 2018 5:03:14 PM

Wrangler will be returning to production shortly however we are not able to restore the gpfs/flash filesystem at this point. This is a hardware related issue that is currently being worked on. Please note that jobs that we can identify as definitely using gpfs/flash have been suspended and will be automatically resumed when the filesystem is available.

Original Posting

Wrangler will be taking an emergency maintenance on Tuesday the 13th to recover from a hardware failure that requires filesystems to be unmounted to resolve. We apologize for the short notice.