Advertisement
v1ral_ITS

Easily safely remove DUPLICATE Files

Jul 2nd, 2018
128
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 9.75 KB | None | 0 0
  1. $ sudo apt-get install fdupes
  2.  
  3. On CentOS/RHEL and Fedora based systems, you need to turn on epel repository to install fdupes package.
  4.  
  5. # yum install fdupes
  6. # dnf install fdupes    [On Fedora 22 onwards]
  7.  
  8. Note: The default package manager yum is replaced by dnf from Fedora 22 onwards…
  9. How to use fdupes command?
  10.  
  11. 1. For demonstration purpose, let’s a create few duplicate files under a directory (say tecmint) simply as:
  12.  
  13. $ mkdir /home/"$USER"/Desktop/tecmint && cd /home/"$USER"/Desktop/tecmint && for i in {1..15}; do echo "I Love Tecmint. Tecmint is a very nice community of Linux Users." > tecmint${i}.txt ; done
  14.  
  15. After running above command, let’s verify the duplicates files are created or not using ls command.
  16.  
  17. $ ls -l
  18. total 60
  19. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint10.txt
  20. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint11.txt
  21. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint12.txt
  22. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint13.txt
  23. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint14.txt
  24. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint15.txt
  25. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint1.txt
  26. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint2.txt
  27. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint3.txt
  28. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint4.txt
  29. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint5.txt
  30. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint6.txt
  31. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint7.txt
  32. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint8.txt
  33. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9.txt
  34.  
  35. The above script create 15 files namely tecmint1.txt, tecmint2.txt…tecmint15.txt and every files contains the same data i.e.,
  36.  
  37. "I Love Tecmint. Tecmint is a very nice community of Linux Users."
  38.  
  39. 2. Now search for duplicate files within the folder tecmint.
  40.  
  41. $ fdupes /home/$USER/Desktop/tecmint
  42. /home/tecmint/Desktop/tecmint/tecmint13.txt
  43. /home/tecmint/Desktop/tecmint/tecmint8.txt
  44. /home/tecmint/Desktop/tecmint/tecmint11.txt
  45. /home/tecmint/Desktop/tecmint/tecmint3.txt
  46. /home/tecmint/Desktop/tecmint/tecmint4.txt
  47. /home/tecmint/Desktop/tecmint/tecmint6.txt
  48. /home/tecmint/Desktop/tecmint/tecmint7.txt
  49. /home/tecmint/Desktop/tecmint/tecmint9.txt
  50. /home/tecmint/Desktop/tecmint/tecmint10.txt
  51. /home/tecmint/Desktop/tecmint/tecmint2.txt
  52. /home/tecmint/Desktop/tecmint/tecmint5.txt
  53. /home/tecmint/Desktop/tecmint/tecmint14.txt
  54. /home/tecmint/Desktop/tecmint/tecmint1.txt
  55. /home/tecmint/Desktop/tecmint/tecmint15.txt
  56. /home/tecmint/Desktop/tecmint/tecmint12.txt
  57.  
  58. 3. Search for duplicates recursively under every directory including it’s sub-directories using the -r option.
  59.  
  60. It search across all the files and folder recursively, depending upon the number of files and folders it will take some time to scan duplicates. In that mean time, you will be presented with the total progress in terminal, something like this.
  61.  
  62. $ fdupes -r /home
  63. Progress [37780/54747] 69%
  64.  
  65. 4. See the size of duplicates found within a folder using the -S option.
  66.  
  67. $ fdupes -S /home/$USER/Desktop/tecmint
  68. 65 bytes each:                          
  69. /home/tecmint/Desktop/tecmint/tecmint13.txt
  70. /home/tecmint/Desktop/tecmint/tecmint8.txt
  71. /home/tecmint/Desktop/tecmint/tecmint11.txt
  72. /home/tecmint/Desktop/tecmint/tecmint3.txt
  73. /home/tecmint/Desktop/tecmint/tecmint4.txt
  74. /home/tecmint/Desktop/tecmint/tecmint6.txt
  75. /home/tecmint/Desktop/tecmint/tecmint7.txt
  76. /home/tecmint/Desktop/tecmint/tecmint9.txt
  77. /home/tecmint/Desktop/tecmint/tecmint10.txt
  78. /home/tecmint/Desktop/tecmint/tecmint2.txt
  79. /home/tecmint/Desktop/tecmint/tecmint5.txt
  80. /home/tecmint/Desktop/tecmint/tecmint14.txt
  81. /home/tecmint/Desktop/tecmint/tecmint1.txt
  82. /home/tecmint/Desktop/tecmint/tecmint15.txt
  83. /home/tecmint/Desktop/tecmint/tecmint12.txt
  84.  
  85. 5. You can see the size of duplicate files for every directory and subdirectories encountered within using the -S and -r options at the same time, as:
  86.  
  87. $ fdupes -Sr /home/avi/Desktop/
  88. 65 bytes each:                          
  89. /home/tecmint/Desktop/tecmint/tecmint13.txt
  90. /home/tecmint/Desktop/tecmint/tecmint8.txt
  91. /home/tecmint/Desktop/tecmint/tecmint11.txt
  92. /home/tecmint/Desktop/tecmint/tecmint3.txt
  93. /home/tecmint/Desktop/tecmint/tecmint4.txt
  94. /home/tecmint/Desktop/tecmint/tecmint6.txt
  95. /home/tecmint/Desktop/tecmint/tecmint7.txt
  96. /home/tecmint/Desktop/tecmint/tecmint9.txt
  97. /home/tecmint/Desktop/tecmint/tecmint10.txt
  98. /home/tecmint/Desktop/tecmint/tecmint2.txt
  99. /home/tecmint/Desktop/tecmint/tecmint5.txt
  100. /home/tecmint/Desktop/tecmint/tecmint14.txt
  101. /home/tecmint/Desktop/tecmint/tecmint1.txt
  102. /home/tecmint/Desktop/tecmint/tecmint15.txt
  103. /home/tecmint/Desktop/tecmint/tecmint12.txt
  104. 107 bytes each:
  105. /home/tecmint/Desktop/resume_files/r-csc.html
  106. /home/tecmint/Desktop/resume_files/fc.html
  107.  
  108. 6. Other than searching in one folder or all the folders recursively, you may choose to choose in two folders or three folders as required. Not to mention you can use option -S and/or -r if required.
  109.  
  110. $ fdupes /home/avi/Desktop/ /home/avi/Templates/
  111.  
  112. 7. To delete the duplicate files while preserving a copy you can use the option ‘-d’. Extra care should be taken while using this option else you might end up loosing necessary files/data and mind it the process is unrecoverable.
  113.  
  114. $ fdupes -d /home/$USER/Desktop/tecmint
  115. [1] /home/tecmint/Desktop/tecmint/tecmint13.txt
  116. [2] /home/tecmint/Desktop/tecmint/tecmint8.txt
  117. [3] /home/tecmint/Desktop/tecmint/tecmint11.txt
  118. [4] /home/tecmint/Desktop/tecmint/tecmint3.txt
  119. [5] /home/tecmint/Desktop/tecmint/tecmint4.txt
  120. [6] /home/tecmint/Desktop/tecmint/tecmint6.txt
  121. [7] /home/tecmint/Desktop/tecmint/tecmint7.txt
  122. [8] /home/tecmint/Desktop/tecmint/tecmint9.txt
  123. [9] /home/tecmint/Desktop/tecmint/tecmint10.txt
  124. [10] /home/tecmint/Desktop/tecmint/tecmint2.txt
  125. [11] /home/tecmint/Desktop/tecmint/tecmint5.txt
  126. [12] /home/tecmint/Desktop/tecmint/tecmint14.txt
  127. [13] /home/tecmint/Desktop/tecmint/tecmint1.txt
  128. [14] /home/tecmint/Desktop/tecmint/tecmint15.txt
  129. [15] /home/tecmint/Desktop/tecmint/tecmint12.txt
  130. Set 1 of 1, preserve files [1 - 15, all]:
  131.  
  132. You may notice that all the duplicates are listed and you are prompted to delete, either one by one or certain range or all in one go. You may select a range something like below to delete files files of specific range.
  133.  
  134. Set 1 of 1, preserve files [1 - 15, all]: 2-15
  135. [-] /home/tecmint/Desktop/tecmint/tecmint13.txt
  136. [+] /home/tecmint/Desktop/tecmint/tecmint8.txt
  137. [-] /home/tecmint/Desktop/tecmint/tecmint11.txt
  138. [-] /home/tecmint/Desktop/tecmint/tecmint3.txt
  139. [-] /home/tecmint/Desktop/tecmint/tecmint4.txt
  140. [-] /home/tecmint/Desktop/tecmint/tecmint6.txt
  141. [-] /home/tecmint/Desktop/tecmint/tecmint7.txt
  142. [-] /home/tecmint/Desktop/tecmint/tecmint9.txt
  143. [-] /home/tecmint/Desktop/tecmint/tecmint10.txt
  144. [-] /home/tecmint/Desktop/tecmint/tecmint2.txt
  145. [-] /home/tecmint/Desktop/tecmint/tecmint5.txt
  146. [-] /home/tecmint/Desktop/tecmint/tecmint14.txt
  147. [-] /home/tecmint/Desktop/tecmint/tecmint1.txt
  148. [-] /home/tecmint/Desktop/tecmint/tecmint15.txt
  149. [-] /home/tecmint/Desktop/tecmint/tecmint12.txt
  150.  
  151. 8. From safety point of view, you may like to print the output of ‘fdupes’ to file and then check text file to decide what file to delete. This decrease chances of getting your file deleted accidentally. You may do:
  152.  
  153. $ fdupes -Sr /home > /home/fdupes.txt
  154.  
  155. Note: You may replace ‘/home’ with the your desired folder. Also use option ‘-r’ and ‘-S’ if you want to search recursively and Print Size, respectively.
  156.  
  157. 9. You may omit the first file from each set of matches by using option ‘-f’.
  158.  
  159. First List files of the directory.
  160.  
  161. $ ls -l /home/$USER/Desktop/tecmint
  162. total 20
  163. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (3rd copy).txt
  164. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (4th copy).txt
  165. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (another copy).txt
  166. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (copy).txt
  167. -rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9.txt
  168.  
  169. and then omit the first file from each set of matches.
  170.  
  171. $ fdupes -f /home/$USER/Desktop/tecmint
  172. /home/tecmint/Desktop/tecmint9 (copy).txt
  173. /home/tecmint/Desktop/tecmint9 (3rd copy).txt
  174. /home/tecmint/Desktop/tecmint9 (another copy).txt
  175. /home/tecmint/Desktop/tecmint9 (4th copy).txt
  176.  
  177. 10. Check installed version of fdupes.
  178.  
  179. $ fdupes --version
  180. fdupes 1.51
  181.  
  182. 11. If you need any help on fdupes you may use switch ‘-h’.
  183.  
  184. $ fdupes -h
  185. Usage: fdupes [options] DIRECTORY...
  186. -r --recurse        for every directory given follow subdirectories
  187. encountered within
  188. -R --recurse:       for each directory given after this option follow
  189. subdirectories encountered within (note the ':' at
  190. the end of the option, manpage for more details)
  191. -s --symlinks       follow symlinks
  192. -H --hardlinks      normally, when two or more files point to the same
  193. disk area they are treated as non-duplicates; this
  194. option will change this behavior
  195. -n --noempty        exclude zero-length files from consideration
  196. -A --nohidden       exclude hidden files from consideration
  197. -f --omitfirst      omit the first file in each set of matches
  198. -1 --sameline       list each set of matches on a single line
  199. -S --size           show size of duplicate files
  200. -m --summarize      summarize dupe information
  201. -q --quiet          hide progress indicator
  202. -d --delete         prompt user for files to preserve and delete all
  203. others; important: under particular circumstances,
  204. data may be lost when using this option together
  205. with -s or --symlinks, or when specifying a
  206. particular directory more than once; refer to the
  207. fdupes documentation for additional information
  208. -N --noprompt       together with --delete, preserve the first file in
  209. each set of duplicates and delete the rest without
  210. prompting the user
  211. -v --version        display fdupes version
  212. -h --help           display this help message
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement