This info is not new*, not my own work, and not super useful right now- can you make it better? Links at bottom.
*ok the bash script is new. I couldn’t stand the thought of having to compile a C function to convert a DATE into HEX…thanks ChatGPT!
Backstory
A client backup NAS needed an update, but we noticed it had been up for 109 days, so a reboot first was in order… and it didn’t come back up!
The issue was possibly this- we had some software installed to monitor the NAS devices, but it gets installed on the ‘boot’ partition of the NAS, and takes up juuuuust enough space so you can’t install updates. Obviously we’d need to remove it before doing the update but now the NAS wouldn’t boot at all, we had a bigger issue. (ok I don’t really think it was just this software, sometimes it’s a combination of 3rd party software and logs. Yes logs…)
To fix this, we’d have to re-install Synology DSM, Can do this via the GUI in ‘finds.synology.com‘ but it barfs and says it can’t do it because the disks are damaged or something. This means we’d need to recover using a reset tool (pin) and either go onsite for the 18 second reset (10+4+4 seconds) or ask a client to do it.
Exploring our options further, we found that when in this state, you can trigger a recovery state and look further into the issue-
Trigger the recovery state
- Attempt to re-install DSM
- Wait for it to fail
- NAS device automatically turns on telnet access
this post also shows you can trigger telnet by visiting this URL- might be more convenient than having to reboot the NAS and wait for failure!
http://<dsm-ip>:5000/webman/start_telnet.cgi
You will get a reply in your web browser saying
{"success": true}So how do you log in?
You’ll have to
- Install telnet on your computer using Homebrew,
- Use this bash script to find the ‘password of the day’. If that doesn’t work, use ‘101-0101’ (which correlates to 1 January) – and this did work for me.
- Login using ‘admin’ or ‘root’ and your shiny new password
In case it’s not obvious, use this syntax –
telnet <DSM-IP>What Now?
Here’s the results of a couple of commands I ran using telnet-
DiskStation> cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdh2[7] sdg2[6] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda2[0]
2097088 blocks [8/8] [UUUUUUUU]
md0 : active raid1 sda1[0] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
2490176 blocks [8/8] [UUUUUUUU]
unused devices: <none>DiskStation> tail /var/log/messages
Jan 1 00:00:00 install.cgi: ninstaller.c:172 Fail to execute [/bin/umount -f > /dev/null 2>&1]
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NE0008-2JM101 EN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NE0008-2JM101 EN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NE0008-2JM101 EN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NE0008-2JM101 EN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NE0008-2JM101 EN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000DM0004-1ZC101 DN01 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NM001G-2MW103 SN03 64570.0
Jan 1 00:00:09 synodiskport: disk/disk_rule_junior.c:109 Failed to get target on ST10000NM001G-2MW103 SN03 64570.0
Jan 1 00:05:01 login[27412]: root login on 'pts/0'You can see from the results here that there’s nothing much wrong, except an installer failed to run because the ‘umount’ command didn’t umount…
What Fixed It?
Due to the lack of serious issues found, I figured a ‘reboot’ was worth a try, and it worked!
To be specific, on boot the NAS decided it needed a new DSM install, allowed me to select ‘keep settings’ and proceeded to install. It took a while to come back up (maybe a disk scrub?) but soon it was happily ingesting M365 data and hungrily waiting for the nightly backups.
Remember I was specifically trying to not to install new DSM, I merely wanted to reboot the device so I could remove OSQuery and THEN upgrade. Oh well, can’t complain about the result!
A reboot didn’t fix it- what now?
All the system data is on the RAID volume ‘md0’ so we need to create a temp directory and mount this volume inside it.
Create temp directory
SynologyNAS> mkdir /mnt/md0Mount md0 into this temp directory
SynologyNAS> mount /dev/md0 /mnt/md0Now find the size of contents-
SynologyNAS> du -sh /mnt/md0/*We deleted some updates and images-
SynologyNAS> rm -rf /mnt/md0/@autoupdateSynologyNAS> rm -rf /mnt/md0/upd@te/Now we need to look at /var
SynologyNAS> du -sh /mnt/md0/var/*
4.0K /mnt/md0/var/cache
4.0K /mnt/md0/var/crash
77.6M /mnt/md0/var/db
4.0K /mnt/md0/var/empty
29.3M /mnt/md0/var/lib
0 /mnt/md0/var/lock
68.1M /mnt/md0/var/log
9.4M /mnt/md0/var/packages
0 /mnt/md0/var/run
12.0K /mnt/md0/var/services
16.0K /mnt/md0/var/spool
4.0K /mnt/md0/var/state
10.4M /mnt/md0/var/synobackup
12.0K /mnt/md0/var/target
4.0K /mnt/md0/var/tmp
20.0K /mnt/md0/var/updateThere might also be some stuff you can delete in /usr
/mnt/md0//var/usr/*Here’s a couple of useful commands-
du -sh /mnt/md0/usr/local/*Got it down to 1.2GB, is that enough?
YES!
SynologyNAS> du -sh /mnt/md0/
1.2G /mnt/md0/Note- you can safely delete @appstore packages, they’ll show as requiring repair when the device boots up, but that might be the space you need(I needed it)!
Resources
Massive thanks to the people who documented these things, you are true legends
This might be the earliest article about the secret telnet password-
https://wrgms.com/synologys-secret-telnet-password/
This one also very interesting about booting in single user mode (not tested)-
https://wrgms.com/entering-single-user-mode-on-a-synology/
This article about enabling telnet without going through the recovery mode steps (not tested)
https://xpenology.com/forum/topic/54694-howto-synology-start-telnetd-service-when-you-want-to-troubleshoot/
This shows results of a ‘help’ command in telnet and lists commands (note it is a Marvell based device and quite old)-
https://conetix.com.au/blog/synology-nas-bootrom-recovery/
This Wayback Archived article shows some of the commands to recover an actual crashed array-
https://web.archive.org/web/20210226133602/http://www.dsebastien.net/2015/05/19/recovering-a-raid-array-in-e-state-on-a-synology-nas/
