ACCRE R9 Cluster Quick and Dirty Status
Report generated at Wed Nov 4 06:23:01 PM CST 2026
Problem Nodes
HOSTNAMES STATE TIMESTAMP REASON COMMENT
cn1325 down* 2026-11-04T12:02:02 Not responding (null)
cn1484 drain* 2026-10-06T15:03:24 Troy - RT101522 - fo Troy - RT101522 - for cn1484 TDX replace fiber breakout
cn1595 drain* 2026-07-27T16:58:32 Troy - RT100528 - Th Troy - RT100528 - Thermal issues
cn1706 drain 2026-11-04T18:22:01 na - na - update-pen na - na - update-pending
cn1817 drain 2026-09-08T11:04:20 Nobody - RT101264 - Nobody - RT101264 - Drain for CGW file recovery
dgx03 inval 2026-11-04T17:41:54 gres/gpu count repor auto - na - update-running
Upgrade Campaign Status
NODE UPGRADE - target 5.14.0-687.64.1
========================================================================
FLEET [#######---------------------] 25% 118/470 on target
(1 partition fully on target, not listed: sam)
batch 69/387 [####----------------] 18%
batch_gpu 17/42 [########------------] 40%
interactive 26/32 [################----] 81%
interactive_gpu 4/7 [###########---------] 57%
------------------------------------------------------------------------
IN THE WORKFLOW 1 node out of service
Unreachable (down / rebooting): 1
interactive_gpu auto 1 dgx03
------------------------------------------------------------------------
LAST 24 HOURS 110 back in service, 0 handed back untouched
Completed: 110
auto 66 cn[1202,1204,1208,1212,1221,1224,1238-1239,1259,1262,1302,1311,1323-1325,1328,1339,1349,1357-1358,1366,1369,1377,1381,1383,1389,1392,1408,1410,1416,1442,1456,1461-1462,1467-1468,1470,1472,1491,1503,1505,1520,1532,1537,1545,1562,1564,1578,1584,1592-1593,1599,1620,1702,1704-1705,1710,1804,1814],gpu[0058,0085,0200-0202,0208,0302]
goffta1 44 cn[1326,1329-1330,1583,1585-1588,1597,1601,1603-1604,1606-1610,1707,1801-1803,1805-1811,1815-1816,1818-1819],dgx04,gpu[0303-0310],gracehopper[01-02],hgx02
Queue Summary (Batch)
GROUP USER ACTIVE_JOBS ACTIVE_CORES PENDING_JOBS PENDING_CORES
-----------------------------------------------------------------------------------------
accre_guests 1 1 1 100
haojz 0 0 1 100
parkc25 1 1 0 0
-----------------------------------------------------------------------------------------
atate_lab 25 50 46 92
aduom 25 50 46 92
-----------------------------------------------------------------------------------------
behringer_lab 2 24 0 0
haleof 2 24 0 0
-----------------------------------------------------------------------------------------
booth_lab_yanjun 2 5 0 0
weiy8 2 5 0 0
-----------------------------------------------------------------------------------------
cem_lab 0 0 192 781
agarwm5 0 0 192 781
-----------------------------------------------------------------------------------------
cgg 0 0 152 848
lib13 0 0 151 784
liy110 0 0 1 64
-----------------------------------------------------------------------------------------
cms 283 4582 485 1522
cmslocal 154 2230 69 243
cmspilot 128 2348 416 1279
uscmslocal 1 4 0 0
-----------------------------------------------------------------------------------------
coxlab 5 27 0 0
blostf1 5 27 0 0
-----------------------------------------------------------------------------------------
das_lab 1 1 0 0
shiltmh1 1 1 0 0
-----------------------------------------------------------------------------------------
davis_lab 0 0 1 16
bluejor 0 0 1 16
-----------------------------------------------------------------------------------------
es3890 0 0 28 224
zahadg 0 0 28 224
-----------------------------------------------------------------------------------------
feng_lab 1 1 0 0
kilics1 1 1 0 0
-----------------------------------------------------------------------------------------
g_gamazon_lab 2 53 0 0
evanspd1 1 3 0 0
wudh 1 50 0 0
-----------------------------------------------------------------------------------------
g_jeffery_lab 1 4 0 0
wilimidj 1 4 0 0
-----------------------------------------------------------------------------------------
gravity 10 40 56 110
ligo 10 40 56 110
-----------------------------------------------------------------------------------------
h_biostat_student 2 17 0 0
namy1 1 1 0 0
yih4 1 16 0 0
-----------------------------------------------------------------------------------------
h_oguz_lab 0 0 4 16
lih30 0 0 4 16
-----------------------------------------------------------------------------------------
h_vmac 27 314 0 0
jiangz8 1 8 0 0
waltes4 1 6 0 0
wangt29 25 300 0 0
-----------------------------------------------------------------------------------------
h_vuiis 50 100 0 0
vuiis_archive 50 100 0 0
-----------------------------------------------------------------------------------------
isde-rer 0 0 1 8
maybergm 0 0 1 8
-----------------------------------------------------------------------------------------
kaczkurkin_lab 1 15 0 0
abbasia 1 15 0 0
-----------------------------------------------------------------------------------------
l2_bioinfo1 3 9 0 0
lyul1 2 6 0 0
zhouy46 1 3 0 0
-----------------------------------------------------------------------------------------
l2_jan_lab 1 7 0 0
olivij1 1 7 0 0
-----------------------------------------------------------------------------------------
l3_ferrell_lab 8 100 259 3108
graciet 8 100 259 3108
-----------------------------------------------------------------------------------------
l3_runnoe_group 51 264 0 0
kaldorme 51 264 0 0
-----------------------------------------------------------------------------------------
l3_vuiis_cci 6 10 0 0
vuiis_daily_s 6 10 0 0
-----------------------------------------------------------------------------------------
lea_lab 44 324 18986 100835
arneram 4 4 18 18
longtial 40 320 9937 79496
songm6 0 0 9031 21321
-----------------------------------------------------------------------------------------
maiziezhou_lab 2 20 1 1
karima1 1 16 1 1
kartha1 1 4 0 0
-----------------------------------------------------------------------------------------
maiziezhou_lab_phd 10 200 0 0
yuanw2 10 200 0 0
-----------------------------------------------------------------------------------------
maple_lab 0 0 1 2
changx3 0 0 1 2
-----------------------------------------------------------------------------------------
mltf 0 0 4 4
mltfpilot 0 0 4 4
-----------------------------------------------------------------------------------------
moro_lab 4 90 0 0
budhirs 1 1 0 0
lix87 2 76 0 0
wangx97 1 13 0 0
-----------------------------------------------------------------------------------------
nasa_imqcam 10 160 11 176
adapav 10 160 11 176
-----------------------------------------------------------------------------------------
nbody 98 161 18796 19307
blakeghg 97 97 18795 18795
hayek13 1 64 0 0
smitm77 0 0 1 512
-----------------------------------------------------------------------------------------
neurogroup 1 5 0 0
roggeokk 1 5 0 0
-----------------------------------------------------------------------------------------
ng_lab 1 16 0 0
kimj119 1 16 0 0
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 315 315
goffta1 0 0 315 315
-----------------------------------------------------------------------------------------
p_cortez_lab 8 128 54 412
johnsae5 8 128 54 412
-----------------------------------------------------------------------------------------
p_csb_meiler 0 0 10785 21014
agarwm5 0 0 10290 20417
yange8 0 0 477 477
zahadg 0 0 18 120
-----------------------------------------------------------------------------------------
p_dsi 0 0 4 6
yangi1 0 0 4 6
-----------------------------------------------------------------------------------------
p_englot_group 59 472 937 7496
redaa1 59 472 937 7496
-----------------------------------------------------------------------------------------
platypus 8 16 319 638
mohamb2 8 16 319 638
-----------------------------------------------------------------------------------------
p_matheny_lab 50 250 0 0
koolajd1 50 250 0 0
-----------------------------------------------------------------------------------------
p_meiler 0 0 1 3
yange8 0 0 1 3
-----------------------------------------------------------------------------------------
rer 2 12 0 0
karomnj 1 8 0 0
theowc 1 4 0 0
-----------------------------------------------------------------------------------------
rke_group 156 343 108 540
griffl6 45 225 108 540
maduren 110 110 0 0
plattecw 1 8 0 0
-----------------------------------------------------------------------------------------
rokaslab 421 487 6040 18367
copea1 1 1 0 0
danist 420 486 6040 18367
-----------------------------------------------------------------------------------------
rubinov_lab 1 40 4 528
wuw11 1 40 4 528
-----------------------------------------------------------------------------------------
salman_lab 1 2 0 0
salmand 1 2 0 0
-----------------------------------------------------------------------------------------
sbcs 1 100 0 0
shij19 1 100 0 0
-----------------------------------------------------------------------------------------
sbgrid 0 0 28 30
agarwm5 0 0 28 30
-----------------------------------------------------------------------------------------
taylor_group 1 3 0 0
petrop3 1 3 0 0
-----------------------------------------------------------------------------------------
vgi 5 69 0 0
actkinkv 1 9 0 0
parkj71 2 44 0 0
tsail2 2 16 0 0
-----------------------------------------------------------------------------------------
walker_lab 1 4 0 0
mathua1 1 4 0 0
-----------------------------------------------------------------------------------------
wankowicz_lab 14 14 0 0
wankows 14 14 0 0
-----------------------------------------------------------------------------------------
womelsdorf_lab 18 72 1 4
gerritcg 18 72 1 4
-----------------------------------------------------------------------------------------
yang_lab 1 1 5 257
larkinsd 1 1 0 0
liuy145 0 0 4 256
resom32 0 0 1 1
-----------------------------------------------------------------------------------------
Totals: 1399 8613 57625 176760
Queue Summary (Batch GPU)
GROUP USER ACTIVE_JOBS ACTIVE_GPUS PENDING_JOBS PENDING_GPUS
-----------------------------------------------------------------------------------------
accre_guests_acc 6 6 5 6
lib13 0 0 1 1
linx10 0 0 1 2
lissm 6 6 0 0
liy110 0 0 3 3
-----------------------------------------------------------------------------------------
cem_lab_gpu_acc 2 3 2 2
agarwm5 0 0 2 2
araizg1 1 1 0 0
holtcm 1 2 0 0
-----------------------------------------------------------------------------------------
cgg_acc 1 1 0 0
liy110 1 1 0 0
-----------------------------------------------------------------------------------------
csb_gpu_acc 1 1 5282 5291
agarwm5 0 0 70 70
buttc 0 0 3 3
gonzc11 0 0 5206 5206
liuy145 1 1 0 0
ranx 0 0 2 8
yangz24 0 0 1 4
-----------------------------------------------------------------------------------------
es3890_acc 10 10 15 15
zahadg 10 10 15 15
-----------------------------------------------------------------------------------------
h_oguz_lab_acc 0 0 2 2
lih30 0 0 2 2
-----------------------------------------------------------------------------------------
maple_lab_acc 0 0 5 5
changx3 0 0 5 5
-----------------------------------------------------------------------------------------
mchaourab_acc 77 77 54 54
lindsr1 77 77 54 54
-----------------------------------------------------------------------------------------
mltf_acc 0 0 1 1
mltfpilot 0 0 1 1
-----------------------------------------------------------------------------------------
nbody_acc 8 8 7 7
bustam1 8 8 7 7
-----------------------------------------------------------------------------------------
noble_lab_acc 2 2 1 1
khodape 0 0 1 1
sud 2 2 0 0
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 25 112
goffta1 0 0 25 112
-----------------------------------------------------------------------------------------
p_dsi_acc 0 0 8 8
yangi1 0 0 8 8
-----------------------------------------------------------------------------------------
p_englot_group_acc 2 2 2 2
zahadg 2 2 2 2
-----------------------------------------------------------------------------------------
taylor_group_acc 3 7 0 0
criswea 1 2 0 0
laaln1 1 4 0 0
schultls 1 1 0 0
-----------------------------------------------------------------------------------------
yang_lab_acc 1 1 0 0
larkinsd 1 1 0 0
-----------------------------------------------------------------------------------------
Totals: 113 118 5409 5506
Queue Summary (interactive)
GROUP USER ACTIVE_JOBS ACTIVE_CORES PENDING_JOBS PENDING_CORES
-----------------------------------------------------------------------------------------
booth_lab_int 3 3 0 0
comptoab 1 1 0 0
mathura 1 1 0 0
vessa 1 1 0 0
-----------------------------------------------------------------------------------------
cem_lab_int 0 0 122 244
agarwm5 0 0 122 244
-----------------------------------------------------------------------------------------
g_giri_group_int 2 8 0 0
giria1 2 8 0 0
-----------------------------------------------------------------------------------------
h_vmac_imaging_int 3 3 0 0
jackb13 3 3 0 0
-----------------------------------------------------------------------------------------
maiziezhou_lab_phd_int 1 128 3 48
zhuy45 1 128 3 48
-----------------------------------------------------------------------------------------
nbody_int 1 1 0 0
pater32 1 1 0 0
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 5 5
goffta1 0 0 5 5
-----------------------------------------------------------------------------------------
rke_group_int 0 0 5 72
kunnar1 0 0 5 72
-----------------------------------------------------------------------------------------
rubinov_lab_int 2 25 2 16
abbasia 1 15 0 0
sardarn 1 10 0 0
wuw11 0 0 2 16
-----------------------------------------------------------------------------------------
vgi_int 1 4 0 0
nitinr 1 4 0 0
-----------------------------------------------------------------------------------------
yang_lab_int 1 8 0 0
shaoq1 1 8 0 0
-----------------------------------------------------------------------------------------
Totals: 14 180 137 385
Queue Summary (interactive_gpu)
GROUP USER ACTIVE_JOBS ACTIVE_GPUS PENDING_JOBS PENDING_GPUS
-----------------------------------------------------------------------------------------
accre_dgx_iacc 1 4 0 0
rubinom 1 4 0 0
-----------------------------------------------------------------------------------------
dsi_dgx_iacc 3 4 4 24
jenkm22 0 0 4 24
may19 1 2 0 0
mohamb2 2 2 0 0
-----------------------------------------------------------------------------------------
l3_humphreys_lab_iacc 1 1 11 11
rayes2 1 1 11 11
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 2 10
goffta1 0 0 2 10
-----------------------------------------------------------------------------------------
ohpen_lab_iacc 1 1 0 0
linx10 1 1 0 0
-----------------------------------------------------------------------------------------
p_matheny_lab_iacc 1 2 0 0
koolajd1 1 2 0 0
-----------------------------------------------------------------------------------------
Totals: 7 12 17 45
Partition Summary
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
interactive up 14-00:00:0 3 mix- cn[1301,1812-1813]
interactive up 14-00:00:0 1 down* cn1325
interactive up 14-00:00:0 1 drain cn1817
interactive up 14-00:00:0 3 mix cn[1287,1302,1326]
interactive up 14-00:00:0 1 alloc cn1800
interactive up 14-00:00:0 23 idle cn[1322-1324,1328-1330,1707,1801-1811,1814-1816,1818-1819]
batch* up 14-00:00:0 1 drain cn1706
batch* up 14-00:00:0 27 mix- cn[1211,1288,1298,1350,1360,1385,1438,1458,1480,1502,1547-1548,1554,1561,1567-1568,1573,1589,1596,1602,1606,1615,1617,1625,1630,1708,2000]
batch* up 14-00:00:0 3 plnd cn[1607-1609]
batch* up 14-00:00:0 2 drain* cn[1484,1595]
batch* up 14-00:00:0 99 mix cn[1221,1239,1258,1265,1270-1271,1275,1277-1279,1291-1292,1297,1303,1309-1310,1315,1320-1321,1332-1333,1336-1338,1340-1341,1345-1348,1353,1364-1367,1375,1377,1380,1397,1411,1417-1418,1422,1424-1426,1435,1443,1456-1457,1462,1466-1467,1477,1483,1500,1508,1528,1531-1532,1545,1550,1555,1565,1574,1577,1583,1585,1587-1588,1592-1594,1597-1601,1603-1605,1612-1613,1618,1620-1621,1623,1626,1629,1631-1633,1700,1702-1705,1709-1710]
batch* up 14-00:00:0 255 alloc cn[1202-1210,1212-1213,1215-1220,1222-1224,1226-1238,1240-1242,1257,1259-1262,1264,1266-1269,1272-1274,1280-1286,1289-1290,1293-1296,1299,1304-1308,1311-1314,1316-1318,1327,1331,1334-1335,1339,1342-1344,1349,1351-1352,1354,1357-1359,1361-1363,1368-1369,1371-1374,1376,1378-1379,1381-1384,1387-1396,1398-1410,1414-1416,1419-1421,1423,1427,1430-1432,1434,1436-1437,1439,1441-1442,1445-1450,1452-1455,1460-1461,1463-1464,1468-1476,1478-1479,1481-1482,1485-1499,1501,1503-1507,1509-1520,1522-1525,1527,1529-1530,1533-1538,1543-1544,1546,1549,1551-1553,1556-1559,1562-1564,1566,1569-1571,1575-1576,1578-1582,1584,1586,1610,1614,1616,1619,1622,1624,1627-1628,1701]
batch_gpu up 14-00:00:0 10 mix- gpu[0068-0069,0071-0073,0076,0078-0080,0300]
batch_gpu up 14-00:00:0 2 mix hgx[02-03]
batch_gpu up 14-00:00:0 18 alloc gpu[0059,0062-0067,0070,0074-0075,0077,0081-0082,0200-0202,0208,0301]
batch_gpu up 14-00:00:0 12 idle gpu[0085,0302-0310],gracehopper[01-02]
interactive_gpu up 14-00:00:0 1 inval dgx03
interactive_gpu up 14-00:00:0 4 mix dgx01,gpu[0058,0060,0207]
interactive_gpu up 14-00:00:0 2 idle dgx04,gpu0061
sam up 2-02:00:00 2 idle cms-sam-[01-02]
reserved inact infinite 0 n/a