• testdisk - Any good READMEs or FAQ or tutorials on recoverig data??

    From Rick Christian@1:135/377 to All on Wed Aug 14 19:26:27 2019
    Hello everybody!

    OK.. I've got a USB disk that worked fine.. then started to have some hissy fits.. wouldn't write during rsync ops...

    So for some reason it seems that it was getting hotter than normal, although its been sitting in that spot for the last 6months or so???

    Anyway... tl;dr

    I put testdisk on a box... and it CAN FIND the partition that is there, that won't mount now... and it CAN SEE THE dirs in the partition so there is hope it
    might recover the data...

    BUT

    I'd like something that walks thru this step by step, that I can review, think on, review , think on review some more...

    I think I am just going to have Tarjay send me another 4TB drive and back up the original and then use this for a test to learn how to recover the partition
    and put back to use at least to read stuff.... and possibly recover another drive that exhibits similar behavior. (Was stoped, unmounted, removed, and now wont remount..)

    So any one got some GOOD for RISK AVERSE persons tutorials, FAQ's etc.. for testdisk?

    I looked at their site and it seems more geared to HD experts vs. I just want my stuff back (nothing critical) person.

    Thanks


    Rick


    ... NOprah!
    --- GoldED+/LNX 1.1.5-b20161221
    * Origin: Vina's Talos Moon Base Alpha (1:135/377)
  • From Bbsing Bbs@1:227/201 to Rick Christian on Thu Aug 15 20:39:00 2019
    Rick Christian wrote to All <=-


    Hello everybody!

    OK.. I've got a USB disk that worked fine.. then started to have some hissy fits.. wouldn't write during rsync ops...

    So for some reason it seems that it was getting hotter than normal, although its been sitting in that spot for the last 6months or so???
    USB thumb drives, that get to hot ... can crush the onboard memory chips, and controller.

    Anyway... tl;dr

    I put testdisk on a box... and it CAN FIND the partition that is there, that won't mount now... and it CAN SEE THE dirs in the partition so
    there is hope it might recover the data...

    BUT

    I'd like something that walks thru this step by step, that I can
    review, think on, review , think on review some more...

    I think I am just going to have Tarjay send me another 4TB drive and
    back up the original and then use this for a test to learn how to
    recover the partition and put back to use at least to read stuff....
    and possibly recover another drive that exhibits similar behavior. (Was stoped, unmounted, removed, and now wont remount..)

    So any one got some GOOD for RISK AVERSE persons tutorials, FAQ's etc.. for testdisk?

    I looked at their site and it seems more geared to HD experts vs. I
    just want my stuff back (nothing critical) person.
    Hi Rick,

    If your drive was part of a lvm volume group, you may be missing the other part of that volume that is preventing you from seeing all the data. IF not then what I attempt to write below may help.

    I'm not sure there is a recovery. USB ssd type drives (non-spinner drives), have a write limit, eventually they just break.
    Now I've not broke a internal or MR2 type ssd yet, but plenty of USB thumb drives, and micro ssd drives, and often times this is earlier than expected end of life so they are still under warranty.

    If your computer is reporting a different size of the drive than manufacture specification, you could have a controller issue, and that could be solved by a controller swap. .. lots more steps for that recovery.

    Here is what I usually do to test if the disk is broke, the easy way.

    Caveat, must use linux, the drive wasn't part of an lvm volume group:

    find the device of the disk. This is usually known when plugging it into the computer, then do the following:

    dmesg |grep sd

    Another easy way is to do the following:

    sudo fdisk -lu

    Both will produce some identifier about your drive. If it doesn't you may have another issue, but from your statements above those two command should work well enough.

    Next ... now understand this is where you must be precise in your command syntax and typing.

    NOTE!!! dd can wipe out all data on your drive if used incorrectly.
    NOTE(2) dd can take a long long time to read a 4TB drive.

    I'll explain what I'm doing and show you and example.
    dd .. used to read the drive directly ignoring any partitions at all.
    using dd I read the device from start to finish. If it can't read the entire device it usually gets stuck or errors out, and this happens with USB ssd type devices right at the memory chip boundaries. Lets say you have a 8 gig USB thumb drive, most often I've seen at the 4GB boundary the drive is broke, or the 2G boundary, ... etc.

    from my command:

    dmesg |grep sd

    I see a device [sdb] Attached SCSI removable disk ..
    I use the dd command as follows:

    dd if=/dev/sdb of=/dev/null bs=512

    eventually you'll get something like (i made up the numbers below):
    11118188+1 records in
    11118188+1 records out
    5000000000 bytes (5.7 GB, ... ) copied ... 22 MB/s
    If the size of 5.7 GB happens to be the size of your device, you probably have a good working drive, if not, you have a broken ssd and recovery is very difficult. Good working drives with ext2,3,4 formats can usually get data back with an: fsck

    dd will be attempting to read your drive device from start to finish. If it finishes really early say after about 2GB when its a 8GB you see it, and it may error out.

    If you use the dd command as follows:

    dd if=/dev/sdb of=/dev/null bs=512 conv=noerror

    and it runs for days and days, you've got it stuck on some spot that is broke on your drive, but .. the last time I've ever used dd on a USB thumb drive, I've never seen this, I've only seen it stop reading prior to reading the entire drive.

    If your dd can read the entire device, then you can used fsck on the drive, and there are many options.
    Also if dd can read the entire device, I usually use dd to make an exact duplicate of my drive, and operate on the image.

    dd if=/dev/sdb of=./sdb.drive.img bs=512 conv=noerror,notrunc

    Then I start operating on the image. If this is an lvm type setup, ... way more things to do,


    For spinner drives .. there is a different process, you can use dd, but other methods work depending on the partitions and format type.



    --- MultiMail/Linux v0.49
    * Origin: Electronic Warfare BBS | telnet:\\bbs.ewbbs.net (1:227/201)
  • From Rick Christian@1:135/377 to Bbsing Bbs on Wed Sep 11 12:41:58 2019

    Hello Bbsing!

    15 Aug 19 20:39, you wrote to me:


    USB thumb drives, that get to hot ... can crush the onboard memory
    chips, and controller.

    Never had that happen with USB thumb drives... but heat is bad for any electronics period.

    If your drive was part of a lvm volume group, you may be missing the

    Nope.

    Standard USB external HD (not a SSD! I don't use or trust that garbage)

    other part of that volume that is preventing you from seeing all the
    data. IF not then what I attempt to write below may help.

    I can SEE THE PARTIOTION in test disk... it just comes up with the Message "Can't mount drive.... " if I plug it in and try to access it like normal use.

    I'm not sure there is a recovery. USB ssd type drives (non-spinner drives), have a write limit, eventually they just break. Now I've not broke a internal or MR2 type ssd yet, but plenty of USB thumb drives,
    and micro ssd drives, and often times this is earlier than expected
    end of life so they are still under warranty.

    This is why I don't use SSD unreliable and write limited.

    If your computer is reporting a different size of the drive than

    Just can't access the drive if I plug it in...

    Here is what I usually do to test if the disk is broke, the easy way.

    Caveat, must use linux,

    Caveat, I don't use any thingBUT LINUX. OK, Z/vm, z/os, s3xx, :) ;) :) but other crap? nope. :)

    the drive wasn't part of an lvm volume group:

    Nope, don't use'm too risky.


    find the device of the disk. This is usually known when plugging it
    into the computer, then do the following:

    You can see that below.

    NOTE!!! dd can wipe out all data on your drive if used incorrectly. NOTE(2) dd can take a long long time to read a 4TB drive.


    dd is too dangerous for use for exactly #1, and #2.


    For spinner drives .. there is a different process, you can use dd,
    but other methods work depending on the partitions and format type.

    What info for real drives? ie:
    9339387.713615] usb 1-3: USB disconnect, device number 6
    [9339387.714581] sd 7:0:0:0: [sdb] Synchronizing SCSI cache
    [9339387.714706] sd 7:0:0:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [9339395.976052] usb 1-3: new high-speed USB device number 10 using ehci_hcd [9339396.186116] scsi8 : usb-storage 1-3:1.0
    [9339397.184886] scsi 8:0:0:0: Direct-Access Seagate BUP BK 0109
    PQ: 0 ANSI: 6
    [9339397.186602] sd 8:0:0:0: Attached scsi generic sg1 type 0

    I guess I am more interested in:

    1) What acn be done via testdisk to use its internal backup/recovery options.... BEFORE I DO

    2) Reset the info, likely the partition info, so that plugging it in and selecing to open in file manager or mounting it other ways works

    I have HUGE RISK AVERSION. Meaning that measure 40x then review, measure 50x more, review.. backup backup backup.. try it...

    So I'd like to use testdisk to suck out the data I can to say either another 4TB drive I attach to the box and then I can review what I've recovered and go from there.. then test out the steps to recover the disk.. ie: do the x,y,z in testdisk to reset it and then see if it is mountable...

    I just can seem to find a GOOD STEP BY STEP EXPLAIN it in full for testdisk , that after I am at the point, below, do, x,y,z to reset things so the drive is mountable...

    Here is some info from testdisk... I have a box with 7.xx testdisk, but where this golded is at is another box, and I quickly attached and ran this to see ... Which based on looking via the testdisk file browser the STUFF IS STILL THERE, the partition table is fubar'd and needs to be reset????

    testdisk.log
    :~$ cat testdisk.log |more


    Wed Aug 14 19:37:32 2019
    Command line: TestDisk

    TestDisk 6.13, Data Recovery Utility, November 2011
    Christophe GRENIER <grenier@cgsecurity.org>
    http://www.cgsecurity.org
    OS: Linux, kernel 3.2.0-79-generic-pae (#115-Ubuntu SMP Thu Mar 12 14:33:25 UTC
    2015) i686
    Compiler: GCC 4.6
    Compilation date: 2012-02-05T07:16:54
    ext2fs lib: 1.42, ntfs lib: 10:0:0, reiserfs lib: none, ewf lib: none
    /dev/sda: LBA, DCO support
    /dev/sda: size 156301488 sectors
    /dev/sda: user_max 156301488 sectors
    /dev/sda: dco 160086528 sectors
    Warning: can't get size for Disk /dev/mapper/control - 0 B - CHS 1 1 1, sector size=512
    Hard disk list
    Disk /dev/sda - 80 GB / 74 GiB - CHS 9729 255 63, sector size=512 - Maxtor 6Y080L0, S/N:Y286W68E, FW:YAR41BW0
    Disk /dev/sdb - 4000 GB / 3726 GiB - CHS 486401 255 63, sector size=512 - Seagate BUP BK, FW:0109

    Partition table type (auto): EFI GPT
    Disk /dev/sdb - 4000 GB / 3726 GiB - Seagate BUP BK
    Partition table type: EFI GPT

    Analyse Disk /dev/sdb - 4000 GB / 3726 GiB - CHS 486401 255 63
    hdr_size=92
    hdr_lba_self=1
    hdr_lba_alt=7814037166 (expected 7814037166)
    hdr_lba_start=34
    hdr_lba_end=7814037133
    hdr_lba_table=2
    hdr_entries=128
    hdr_entsz=128
    check_part_gpt failed for partition
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    Current partition structure:
    No FAT, NTFS, ext2, JFS, Reiser, cramfs or XFS marker
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    2 P MS Data 264192 7814035455 7813771264 [Basic data partition] [Seagate Backup Plus Drive]

    search_part()
    Disk /dev/sdb - 4000 GB / 3726 GiB - CHS 486401 255 63
    NTFS at 16/113/34
    filesystem size 7813771264
    sectors_per_cluster 8
    mft_lcn 786432
    mftmirr_lcn 2
    clusters_per_mft_record -10
    clusters_per_index_record 1
    MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB
    Search for partition aborted

    Results
    P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB

    interface_write()
    1 P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    simulate write!

    Analyse Disk /dev/sdb - 4000 GB / 3726 GiB - CHS 486401 255 63
    hdr_size=92
    hdr_lba_self=1
    hdr_lba_alt=7814037166 (expected 7814037166)
    hdr_lba_start=34
    hdr_lba_end=7814037133
    hdr_lba_table=2
    hdr_entries=128
    hdr_entsz=128
    check_part_gpt failed for partition
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    Current partition structure:
    No FAT, NTFS, ext2, JFS, Reiser, cramfs or XFS marker
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    1 P MS Reserved 34 262177 262144 [Microsoft reserved
    partition]
    2 P MS Data 264192 7814035455 7813771264 [Basic data partition] [Seagate Backup Plus Drive]

    search_part()
    Disk /dev/sdb - 4000 GB / 3726 GiB - CHS 486401 255 63
    NTFS at 16/113/34
    filesystem size 7813771264
    sectors_per_cluster 8
    mft_lcn 786432
    mftmirr_lcn 2
    clusters_per_mft_record -10
    clusters_per_index_record 1
    MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB

    Results
    P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB
    ntfs_device_testdisk_io_ioctl() unimplemented
    ntfs_device_testdisk_io_ioctl() unimplemented
    NTFS filesystem need to be repaired.


    dir_partition inode=5
    P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB
    Directory /
    5 dr-xr-xr-x 0 0 0 11-Aug-2019 09:01 .
    5 dr-xr-xr-x 0 0 0 11-Aug-2019 09:01 ..
    35 -r--r--r-- 0 0 386470 19-Sep-2018 04:00 .VolumeIcon.icns
    36 -r--r--r-- 0 0 194586 19-Sep-2018 04:00 .VolumeIcon.ico
    37 -r--r--r-- 0 0 4096 19-Sep-2018 04:00 ._
    38 -r--r--r-- 0 0 33 19-Sep-2018 04:00 Autorun.inf
    89 dr-xr-xr-x 0 0 0 12-Aug-2019 22:05 Grabbed_Photos
    384686 dr-xr-xr-x 0 0 0 5-Jun-2019 07:19 PossImages
    41 dr-xr-xr-x 0 0 0 17-Dec-2018 00:07 Seagate
    86 dr-xr-xr-x 0 0 0 11-Aug-2019 10:33 seed
    87 dr-xr-xr-x 0 0 0 11-Aug-2019 19:30 seed_imagesets
    88 dr-xr-xr-x 0 0 0 12-Aug-2019 06:00 Smxxty
    43 dr-xr-xr-x 0 0 0 17-Dec-2018 00:07 Start_Here_Mac.app
    39 -r--r--r-- 0 0 18033016 19-Sep-2018 04:01 Start_Here_Win.exe
    84 dr-xr-xr-x 0 0 0 17-Dec-2018 00:16 System Volume Information
    1422010 dr-xr-xr-x 0 0 0 11-Aug-2019 09:02 testssh
    40 -r--r--r-- 0 0 1172289 19-Sep-2018 04:00 Warranty.pdf

    dir_partition inode=384686
    P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB
    Directory /PossImages
    384686 dr-xr-xr-x 0 0 0 5-Jun-2019 07:19 .
    5 dr-xr-xr-x 0 0 0 11-Aug-2019 09:01 ..
    893834 dr-xr-xr-x 0 0 0 12-Aug-2019 09:42 SxxyBacks
    651926 dr-xr-xr-x 0 0 0 1-May-2019 11:38 SxxyBacks050119
    384688 dr-xr-xr-x 0 0 0 5-Jun-2019 07:17 SxxBacks060519

    dir_partition inode=893834
    P MS Data 264192 7814035455 7813771264 [Seagate Backup Plus Drive]
    NTFS, 4000 GB / 3725 GiB
    ntfs_readdir

    Rick


    ... NOprah!
    --- GoldED+/LNX 1.1.5-b20161221
    * Origin: Vina's Talos Moon Base Alpha (1:135/377)
  • From Dan Clough@1:123/115 to Rick Christian on Wed Sep 11 15:26:00 2019
    Rick Christian wrote to Bbsing Bbs <=-

    I'm not sure there is a recovery. USB ssd type drives (non-spinner drives), have a write limit, eventually they just break. Now I've not broke a internal or MR2 type ssd yet, but plenty of USB thumb drives,
    and micro ssd drives, and often times this is earlier than expected
    end of life so they are still under warranty.

    This is why I don't use SSD unreliable and write limited.

    Your information on SSD's is quite outdated. SSD's are *FAR* from
    unreliable and the "write limited" part is not really a factor any
    more. Even consumer-grade SSD's have an expected lifetime that is
    measured in decades (of far heavier use than you would do).

    NOTE!!! dd can wipe out all data on your drive if used incorrectly. NOTE(2) dd can take a long long time to read a 4TB drive.

    dd is too dangerous for use for exactly #1, and #2.

    Too dangerous? Well, you could get hit by a bus when you cross
    the street, too.

    I have HUGE RISK AVERSION. Meaning that measure 40x then review,
    measure 50x more, review.. backup backup backup.. try it...

    Maybe that's a little excessive?


    ... Internal Error: The system has been taken over by sheep at line 19960
    === MultiMail/Linux v0.52
    --- SBBSecho 3.09-Linux
    * Origin: Palantir * palantirbbs.ddns.net * Pensacola, FL * (1:123/115)
  • From Tony Langdon@3:633/410 to Dan Clough on Thu Sep 12 13:52:00 2019
    On 09-11-19 15:26, Dan Clough wrote to Rick Christian <=-

    Your information on SSD's is quite outdated. SSD's are *FAR* from unreliable and the "write limited" part is not really a factor any
    more. Even consumer-grade SSD's have an expected lifetime that is measured in decades (of far heavier use than you would do).

    Yes, SSDs do have write limitations, but for all practical purposes, modern SSDs are likely to outlive the rest of the system they're installed in. I love them, and will happily use a SSD for the performance gains on the OS drive. A spinning HDD is still handy for bulk data storage though, because they're far cheaper in larger sizes.


    ... But the FACTS keep interfering with your theories!
    === MultiMail/Win v0.51
    --- SBBSecho 3.03-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (3:633/410)
  • From Gerrit Kuehn@2:240/12 to Dan Clough on Thu Sep 12 20:55:36 2019
    Hello Dan!

    11 Sep 19 15:26, Dan Clough wrote to Rick Christian:

    This is why I don't use SSD unreliable and write limited.

    Your information on SSD's is quite outdated. SSD's are *FAR* from unreliable and the "write limited" part is not really a factor any
    more. Even consumer-grade SSD's have an expected lifetime that is measured in decades (of far heavier use than you would do).

    This heavily depends on the use case. I already managed to blow away like 10% of several SSD's expected lifetime in a rather simple home setup within only a couple of months. Write amplification and aggressive caching by webbrowsers like Firefox and Chrome make it possible.


    Regards,
    Gerrit

    ... 8:55PM up 149 days, 5:16, 8 users, load averages: 0.68, 0.50, 0.40

    --- Msged/BSD 6.1.2
    * Origin: All carefully conceived (2:240/12)
  • From Kai Richter@2:240/77 to Rick Christian on Mon Sep 16 00:19:54 2019
    Hello Rick!

    11 Sep 19, Rick Christian wrote to Bbsing Bbs:

    Standard USB external HD (not a SSD! I don't use or trust that
    garbage)

    This is why I don't use SSD unreliable and write limited.

    Reality Check: My SSD works fine, what is broken on your side?
    HDD are unreliable and are write limited just like any other storage media too. They are sold with a MTBF with the warranty that they will fail.

    In 20 years i lost two internal HDD, had two new HDD with many badblocks (replaced by warranty), one dead SSD and two sudden death USB Sticks.

    Bashing one type of storage may calm your emotions, but it would not help you.

    I can SEE THE PARTIOTION in test disk...

    You don't. You see the data entry in the partition table which tells you where the real partition was.

    it just comes up with the Message "Can't mount drive.... " if I plug
    it in and try to access it like normal use.

    If your computer is reporting a different size of the drive than

    Just can't access the drive if I plug it in...

    How could you see something with testdisk then?

    NOTE!!! dd can wipe out all data on your drive if used
    incorrectly. NOTE(2) dd can take a long long time to read a 4TB
    drive.

    dd is too dangerous for use for exactly #1, and #2.

    If you are talking about too dangerous and risky you must use dd to reduce risk
    and danger. There is no way around.

    If you are working with the original data there is a high risk that any rescue tool make things worse or destroy mandatory data.

    At the moment you don't know what kind of damage you have. Is there a software failure in the data structure on the disk or do you have a real hardware failure that prevents some kind of data to be read.

    The first thing is to make a mirror of the damaged device. dd_rescue is a useful Linux tool for that. If testdisk finds something, than some data can be read. dd_rescue copies all data that can be read to another disk and skips the unreadable data. It's suggested to keep the first mirror and make a working copy of that mirror for trouble shooting. A broken disk could loose more data with every usage. Creating additional mirrors from the original disk may result
    into additional data loss.

    1) What acn be done via testdisk to use its internal backup/recovery options.... BEFORE I DO

    It can recover the partition table, the partition directory, from a backup sector. It can't repair a broken filesystem i.e. if the filesystem can't be mounted.

    2) Reset the info, likely the partition info, so that plugging it in
    and selecing to open in file manager or mounting it other ways works

    Can't work. A partition can hold one of dozens of different file systems which are mounted in different ways. Linux also does not need a partition, a filesystem can be placed directly on /dev/sdx.

    I have HUGE RISK AVERSION. Meaning that measure 40x then review,
    measure 50x more, review.. backup backup backup.. try it...

    Hm, sounds like you overkilled your disk with too many backups.

    So I'd like to use testdisk to suck out the data I can to say either another 4TB drive I attach to the box and then I can review what I've recovered and go from there.. then test out the steps to recover the disk..

    Sounds good. Step one: Suck out data = use dd_rescue

    ie: do the x,y,z in testdisk to reset it and then see if it is mountable...

    I just can seem to find a GOOD STEP BY STEP EXPLAIN it in full for testdisk , that after I am at the point, below, do, x,y,z to reset
    things so the drive is mountable...

    There are too many if then else in the process to write them down in a step by step tutorial. Many things are useless and do not apply if a previous "else" matches.

    testdisk is for partitions and testdisk tells you:

    NTFS filesystem need to be repaired.

    that your filesystem is broken. So testdisk is useless.

    I don't know how reliable the linux ntfs repair modes are, if your mirror disk is ready you maybe could give an MS system a try.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Phillip L Taylor Jr@1:275/201.30 to Dan Clough on Tue Nov 3 22:47:45 2020
    On Wed 11-Sep-2019 10:26 , Dan Clough@1:123/115.0 said to Rick Christian:

    I have HUGE RISK AVERSION. Meaning that measure 40x then review, measure 50x more, review.. backup backup backup.. try it...

    Maybe that's a little excessive?

    In the old days they called a mouse a mice.
    --- CNet/5
    * Origin: 1:275/201.0 (1:275/201.30)
  • From Benny Pedersen@2:230/0 to Phillip L Taylor Jr on Fri Nov 6 03:50:44 2020
    Hello Phillip!

    03 Nov 2020 22:47, Phillip L Taylor Jr wrote to Dan Clough:

    I have HUGE RISK AVERSION. Meaning that measure 40x then review,
    measure 50x more, review.. backup backup backup.. try it...

    Maybe that's a little excessive?

    PLTJ> In the old days they called a mouse a mice.

    and the diffrent is that only one could use PS2 port ? :)


    Regards Benny

    ... too late to die young :)

    --- Msged/LNX 6.1.2 (Linux/5.9.6-gentoo-x86_64 (x86_64))
    * Origin: I will always keep a PC running CPM 3.0 (2:230/0)