View difference between Paste ID: AKiC3EhQ and WEDwpcz9
SHOW: | | - or go back to the newest paste.
1-
##################################
1+
#################
2-
# Pentester Academy Log Analysis #
2+
#  Log Analysis #
3-
##################################
3+
#################
4
5-
I'm doing this set of videos for my good friend Vivek Ramachandran at SecurityTube.net/PentesterAcademy.com
5+
6
##########
7
# VMWare #
8
##########
9
- For this workshop you'll need the latest version of VMWare Workstation (Windows), Fusion (Mac), or Player.
10
 
11
- Although you can get the VM to run in VirtualBox, I will not be supporting this configuration for this class.
12
13
14
VM for these labs
15
-----------------
16
https://s3.amazonaws.com/StrategicSec-VMs/StrategicsecUbuntu-v3.zip
17
user: strategicsec
18
pass: strategicsec
19
20-
https://s3.amazonaws.com/infosecaddictsvirtualmachines/Win7x64.zip
20+
https://s3.amazonaws.com/StrategicSec-VMs/Win7x64.zip
21
username: workshop
22
password: password
23
24
25
26
27
##############################################
28
# Log Analysis with Linux command-line tools #
29
##############################################
30
The following command line executables are found in the Mac as well as most Linux Distributions.
31
32
cat –  prints the content of a file in the terminal window
33
grep – searches and filters based on patterns
34
awk –  can sort each row into fields and display only what is needed
35
sed –  performs find and replace functions
36
sort – arranges output in an order
37
uniq – compares adjacent lines and can report, filter or provide a count of duplicates
38
39
40
##############
41
# Cisco Logs #
42
##############
43
44
-----------------------------Type this-----------------------------------------
45
wget https://s3.amazonaws.com/infosecaddictsfiles/cisco.log
46
-------------------------------------------------------------------------------
47
48
AWK Basics
49
----------
50
To quickly demonstrate the print feature in awk, we can instruct it to show only the 5th word of each line. Here we will print $5. Only the last 4 lines are being shown for brevity.
51
52
-----------------------------Type this-----------------------------------------
53
cat cisco.log | awk '{print $5}' | tail -n 4
54
-------------------------------------------------------------------------------
55
56
57
58
Looking at a large file would still produce a large amount of output. A more useful thing to do might be to output every entry found in “$5”, group them together, count them, then sort them from the greatest to least number of occurrences. This can be done by piping the output through “sort“, using “uniq -c” to count the like entries, then using “sort -rn” to sort it in reverse order.
59
60
-----------------------------Type this-----------------------------------------
61
cat cisco.log | awk '{print $5}'| sort | uniq -c | sort -rn
62
-------------------------------------------------------------------------------
63
64
65
66
While that’s sort of cool, it is obvious that we have some garbage in our output. Evidently we have a few lines that aren’t conforming to the output we expect to see in $5. We can insert grep to filter the file prior to feeding it to awk. This insures that we are at least looking at lines of text that contain “facility-level-mnemonic”.
67
68
-----------------------------Type this-----------------------------------------
69
cat cisco.log | grep %[a-zA-Z]*-[0-9]-[a-zA-Z]* | awk '{print $5}' | sort | uniq -c | sort -rn
70
-------------------------------------------------------------------------------
71
72
73
74
75
Now that the output is cleaned up a bit, it is a good time to investigate some of the entries that appear most often. One way to see all occurrences is to use grep.
76
77
-----------------------------Type this-----------------------------------------
78
cat cisco.log | grep %LINEPROTO-5-UPDOWN:
79
80
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| awk '{print $10}' | sort | uniq -c | sort -rn
81
82
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10}' | sort | uniq -c | sort -rn
83
84
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10 " changed to " $14}' | sort | uniq -c | sort -rn
85
--------------------------------------------------------------------------------
86
87
###############
88
# Apache Logs #
89
###############
90
91
Reference:
92
http://www.the-art-of-web.com/system/logs/
93
94
wget https://s3.amazonaws.com/SecureNinja/Python/access_log
95
96
97
You want to list all user agents ordered by the number of times they appear (descending order):
98
99
awk -F\" '{print $6}' access_log | sort | uniq -c | sort -fr
100
101
102
103
Using the default separator which is any white-space (spaces or tabs) we get the following:
104
105
awk '{print $1}' access_log         # ip address (%h)
106
awk '{print $2}' access_log         # RFC 1413 identity (%l)
107
awk '{print $3}' access_log         # userid (%u)
108
awk '{print $4,5}' access_log       # date/time (%t)
109
awk '{print $9}' access_log         # status code (%>s)
110
awk '{print $10}' access_log        # size (%b)
111
112
You might notice that we've missed out some items. To get to them we need to set the delimiter to the " character which changes the way the lines are 'exploded' and allows the following:
113
114
awk -F\" '{print $2}' access_log    # request line (%r)
115
awk -F\" '{print $4}' access_log    # referer
116
awk -F\" '{print $6}' access_log    # user agent
117
118
119
awk -F\" '{print $6}' access_log \
120
  | sed 's/(\([^;]\+; [^;]\+\)[^)]*)/(\1)/' \
121
  | sort | uniq -c | sort -fr
122
123
124-
wget https://s3.amazonaws.com/StrategicSec-Files/LogAnalysis/cisco.log
124+
125
126
awk -F\" '($6 ~ /Googlebot/){print $2}' access_log | awk '{print $2}'
127
Or who's been looking at your guestbook?
128
129
awk -F\" '($2 ~ /guestbook\.html/){print $6}' access_log
130
131
132
Reference:
133
https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/
134
135
# top 20 URLs from the last 5000 hits
136
tail -5000 ./access_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
137
tail -5000 ./access_log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
138
 
139
# top 20 URLS excluding POST data from the last 5000 hits
140
tail -5000 ./access_log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
141
tail -5000 ./access_log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
142
 
143
# top 20 IPs from the last 5000 hits
144
tail -5000 ./access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
145
tail -5000 ./access_log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
146
 
147
# top 20 URLs requested from a certain ip from the last 5000 hits
148
IP=1.2.3.4; tail -5000 ./access_log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
149
IP=1.2.3.4; tail -5000 ./access_log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
150
 
151
# top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
152
IP=1.2.3.4; tail -5000 ./access_log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
153
IP=1.2.3.4; tail -5000 ./access_log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
154
 
155
# top 20 referrers from the last 5000 hits
156
tail -5000 ./access_log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
157
tail -5000 ./access_log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
158
 
159
# top 20 user agents from the last 5000 hits
160
tail -5000 ./access_log | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20
161
 
162
# sum of data (in MB) transferred in the last 5000 hits
163
tail -5000 ./access_log | awk '{sum+=$10} END {print sum/1048576}'
164
165
166
167
168
169
170
171
#################################
172
# Using Python for log analysis #
173
#################################
174
175
python
176
177
>>>
178
179
180
181
###########################################
182
# Python Basics Lesson 1: Simple Printing #
183
###########################################
184
185
>>> print 1
186
187
>>> print hello
188
189
>>> print "hello"
190
 
191
>>> print "Today we are learning Python."
192
 
193
 
194
 
195
###################################################
196
# Python Basics Lesson 2: Simple Numbers and Math #
197
###################################################
198
 
199
>>> 2+2
200
 
201
>>> 6-3
202
 
203
>>> 18/7
204
 
205
>>> 18.0/7
206
 
207
>>> 18.0/7.0
208
 
209
>>> 18/7
210
 
211
>>> 9%4
212
 
213
>>> 8%4
214
 
215
>>> 8.75%.5
216
 
217
>>> 6.*7
218
 
219
>>> 6*6*6
220
 
221
>>> 6**3
222
 
223
>>> 5**12
224
 
225
>>> -5**4
226
 
227
 
228
 
229
 
230
 
231
 
232
#####################################
233
# Python Basics Lesson 3: Variables #
234
#####################################
235
 
236
>>> x=18
237
 
238
>>> x+15
239
 
240
>>> x**3
241
 
242
>>> y=54
243
 
244
>>> x+y
245
 
246
>>> age=input("Enter number here: ")
247
        43
248
 
249
>>> age+32
250
 
251
>>> age**3
252
253
>>> fname = raw_input("Enter your first name: ")
254
255
>>> lname = raw_input("Enter your first name: ")
256
257
>>> fname = raw_input("Enter your name: ")
258
Enter your name: Joe
259
260
>>> lname = raw_input("Enter your name: ")
261
Enter your name: McCray
262
263
>>> print fname
264
Joe
265
266
>>> print lname
267
McCray
268
269
>>> print fname lname
270
271
>>> print fname+lname
272
JoeMcCray
273
274
 
275
 
276
NOTE:
277
Use "input() for integers and expressions, and use raw_input() when you are dealing with strings. 
278
 
279
 
280
 
281
 
282
 
283
#################################################
284
# Python Basics Lesson 4: Modules and Functions #
285
#################################################
286
 
287
>>> 5**4
288
 
289
>>> pow(5,4)
290
 
291
>>> abs(-18)
292
 
293
>>> abs(5)
294
 
295
>>> floor(18.7)
296
 
297
>>> import math
298
 
299
>>> math.floor(18.7)
300
 
301
>>> math.sqrt(81)
302
 
303
>>> joe = math.sqrt
304
 
305
>>> joe(9)
306
 
307
>>> joe=math.floor
308
 
309
>>> joe(19.8)
310
 
311
 
312
 
313
 
314
 
315
 
316
 
317
 
318
 
319
###################################
320
# Python Basics Lesson 5: Strings #
321
###################################
322
 
323
>>> "XSS"
324
 
325
>>> 'SQLi'
326
 
327
>>> "Joe's a python lover"
328
 
329
>>> 'Joe\'s a python lover'
330
 
331
>>> "Joe said \"InfoSec is fun\" to me"
332
 
333
>>> a = "Joe"
334
 
335
>>> b = "McCray"
336
 
337
>>> a, b
338
 
339
>>> a+b
340
 
341
 
342
 
343
 
344
 
345
 
346
 
347
 
348
########################################
349
# Python Basics Lesson 6: More Strings #
350
########################################
351
 
352
>>> num = 10
353
 
354
>>> num + 2
355
 
356
>>> "The number of open ports found on this system is " + num
357
 
358
>>> num = str(18)
359
 
360
>>> "There are " + num + " vulnerabilities found in this environment."
361
 
362
>>> num2 = 46
363
 
364
>>> "As of 08/20/2012, the number of states that enacted the Security Breach Notification Law is " + `num2`
365
 
366
367
 
368
NOTE:
369
Use "input() for integers and expressions, and use raw_input() when you are dealing with strings.
370
 
371
 
372
 
373
 
374
 
375
 
376
 
377
###############################################
378
# Python Basics Lesson 7: Sequences and Lists #
379
###############################################
380
 
381
>>> attacks = ['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
382
 
383
>>> attacks
384
['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
385
 
386
>>> attacks[3]
387
'SQL Injection'
388
 
389
>>> attacks[-2]
390
'Cross-Site Scripting'
391
 
392
 
393
 
394
 
395
 
396
 
397
########################################
398
# Python Basics Level 8: If Statement #
399
########################################
400
>>> attack="SQLI"
401
>>> if attack=="SQLI":
402
        print 'The attacker is using SQLI'
403
 
404
>>> attack="XSS"
405
>>> if attack=="SQLI":
406
        print 'The attacker is using SQLI'
407
408
409
410
411
>>> exit()
412
 
413
#############################
414
# Reference Videos To Watch #
415
#############################
416
Here is your first set of youtube videos that I'd like for you to watch:
417
https://www.youtube.com/playlist?list=PLEA1FEF17E1E5C0DA (watch videos 1-10)
418
 
419
 
420
 
421
 
422
 
423
#####################################
424
# Lesson 9: Intro to Log Analysis #
425
#####################################
426
 
427
Login to your StrategicSec Ubuntu machine. You can download the VM from the following link:
428
 
429
https://s3.amazonaws.com/StrategicSec-VMs/Strategicsec-Ubuntu-VPN-163.zip
430
        username: strategicsec
431
        password: strategicsec
432
 
433
Then execute the following commands:
434
---------------------------------------------------------------------------------------------------------
435
 
436
 
437
wget https://s3.amazonaws.com/SecureNinja/Python/access_log
438
 
439
 
440
cat access_log | grep 141.101.80.188
441
 
442
cat access_log | grep 141.101.80.187
443
 
444
cat access_log | grep 108.162.216.204
445
 
446
cat access_log | grep 173.245.53.160
447
 
448
---------------------------------------------------------
449
 
450
Google the following terms:
451
        - Python read file
452
        - Python read line
453
        - Python read from file
454
 
455
 
456
 
457
 
458
########################################################
459
# Lesson 10: Use Python to read in a file line by line #
460
########################################################
461
 
462
 
463
Reference:
464
http://cmdlinetips.com/2011/08/three-ways-to-read-a-text-file-line-by-line-in-python/
465
466
467
468
469
470
471
Let's have some fun.....
472
473
474
>>> f = open('access_log', "r")
475
476
>>> lines = f.readlines()
477
478
>>> print lines
479
480
>>> lines[0]
481
482
>>> lines[10]
483
484
>>> lines[50]
485
486
>>> lines[1000]
487
488
>>> lines[5000]
489
490
>>> lines[10000]
491
492
>>> print len(lines)
493
494
495
496
497
498
499
 
500
 
501
 
502
---------------------------------------------------------
503
vi logread1.py
504
 
505
 
506
## Open the file with read only permit
507
f = open('access_log', "r")
508
 
509
## use readlines to read all lines in the file
510
## The variable "lines" is a list containing all lines
511
lines = f.readlines()
512
 
513
print lines
514
 
515
 
516
## close the file after reading the lines.
517
f.close()
518
 
519
---------------------------------------------------------
520
 
521
 
522
Google the following:
523
        - python difference between readlines and readline
524
        - python readlines and readline
525
 
526
 
527
 
528
 
529
 
530
#################################
531
# Lesson 11: A quick challenge #
532
#################################
533
 
534
Can you write an if/then statement that looks for this IP and print "Found it"?
535
 
536
 
537
141.101.81.187
538
 
539
 
540
 
541
 
542
 
543
 
544
---------------------------------------------------------
545
Hint 1: Use Python to look for a value in a list
546
 
547
Reference:
548
http://www.wellho.net/mouth/1789_Looking-for-a-value-in-a-list-Python.html
549
 
550
 
551
 
552
 
553
---------------------------------------------------------
554
Hint 2: Use Python to prompt for user input
555
 
556
Reference:
557
http://www.cyberciti.biz/faq/python-raw_input-examples/
558
 
559
 
560
 
561
 
562
---------------------------------------------------------
563
Hint 3: Use Python to search for a string in a list
564
 
565
Reference:
566
http://stackoverflow.com/questions/4843158/check-if-a-python-list-item-contains-a-string-inside-another-string
567
 
568
 
569
 
570
 
571
 
572
Here is my solution:
573
-------------------
574
$ python
575
>>> f = open('access_log', "r")
576
>>> lines = f.readlines()
577
>>> ip = '141.101.81.187'
578
>>> for string in lines:
579
...     if ip in string:
580
...             print(string)
581
 
582
 
583
 
584
 
585
Here is one student's solution - can you please explain each line of this code to me?
586
-------------------------------------------------------------------------------------
587
#!/usr/bin/python
588
 
589
f = open('access_log')
590
 
591
strUsrinput = raw_input("Enter IP Address: ")
592
 
593
for line in iter(f):
594
    ip = line.split(" - ")[0]
595
    if ip == strUsrinput:
596
        print line
597
 
598
f.close()
599
 
600
 
601
 
602
 
603
-------------------------------
604
 
605
Working with another student after class we came up with another solution:
606
 
607
#!/usr/bin/env python
608
 
609
 
610
# This line opens the log file
611
f=open('access_log',"r")
612
 
613
# This line takes each line in the log file and stores it as an element in the list
614
lines = f.readlines()
615
 
616
 
617
# This lines stores the IP that the user types as a var called userinput
618
userinput = raw_input("Enter the IP you want to search for: ")
619
 
620
 
621
 
622
# This combination for loop and nested if statement looks for the IP in the list called lines and prints the entire line if found.
623
for ip in lines:
624
    if ip.find(userinput) != -1:
625
        print ip
626
 
627
 
628
 
629
##################################################
630
# Lesson 12: Look for web attacks in a log file #
631
##################################################
632
 
633
In this lab we will be looking at the scan_log.py script and it will scan the server log to find out common hack attempts within your web server log.
634
Supported attacks:
635
1.          SQL Injection
636
2.          Local File Inclusion
637
3.          Remote File Inclusion
638
4.          Cross-Site Scripting
639
 
640
 
641
 
642
wget https://s3.amazonaws.com/SecureNinja/Python/scan_log.py
643
 
644-
https://s3.amazonaws.com/infosecaddictsvirtualmachines/Win7x64.zip
644+
645
 
646
cat scan_log.py | less                  (use your up/down arrow keys to look through the file)
647
648
649
650
651
652
################################
653
# Log Analysis with Powershell #
654
################################
655
656
VM for these labs
657
-----------------
658
https://s3.amazonaws.com/StrategicSec-VMs/Win7x64.zip
659
        username: workshop
660
        password: password
661
662
 
663
You can do the updates in the Win7 VM (yes, it is a lot of updates).
664
 
665
You'll need to create directory in the Win7 VM called "c:\ps"
666
 
667
#####################
668
# Powershell Basics #
669
#####################
670
 
671
PowerShell is Microsoft’s new scripting language that has been built in since the release Vista.
672
 
673
PowerShell file extension end in .ps1 .
674
 
675
An important note is that you cannot double click on a PowerShell script to execute it.
676
 
677
To open a PowerShell command prompt either hit Windows Key + R and type in PowerShell or Start -> All Programs -> Accessories -> Windows PowerShell -> Windows PowerShell.
678
 
679
dir
680
cd
681
ls
682
cd c:\
683
 
684
 
685
To obtain a list of cmdlets, use the Get-Command cmdlet
686
 
687
Get-Command
688
 
689
 
690
 
691
You can use the Get-Alias cmdlet to see a full list of aliased commands.
692
 
693
Get-Alias
694
 
695
 
696
 
697
Don't worry you won't blow up your machine with Powershell
698
Get-Process | stop-process                              What will this command do?
699
Get-Process | stop-process -whatif
700
 
701
 
702
To get help with a cmdlet, use the Get-Help cmdlet along with the cmdlet you want information about.
703
 
704
Get-Help Get-Command
705
 
706
Get-Help Get-Service –online
707
 
708
Get-Service -Name TermService, Spooler
709
 
710
Get-Service –N BITS
711
 
712
Start-Transcript
713
 
714
PowerShell variables begin with the $ symbol. First lets create a variable
715
 
716
$serv = Get-Service –N Spooler
717
 
718
To see the value of a variable you can just call it in the terminal.
719
 
720
$serv
721
 
722
$serv.gettype().fullname
723
 
724
 
725
Get-Member is another extremely useful cmdlet that will enumerate the available methods and properties of an object. You can pipe the object to Get-Member or pass it in
726
 
727
$serv | Get-Member
728
 
729
Get-Member -InputObject $serv
730
 
731
 
732
 
733
 
734
 
735
Let’s use a method and a property with our object.
736
 
737
$serv.Status
738
$serv.Stop()
739
$serv.Refresh()
740
$serv.Status
741
$serv.Start()
742
$serv.Refresh()
743
$serv.Status
744
 
745
 
746
 
747
 
748
Methods can return properties and properties can have sub properties. You can chain them together by appending them to the first call.
749
 
750
 
751
 
752
#############################
753
# Simple Event Log Analysis #
754
#############################
755
 
756
Step 1: Dump the event logs
757
---------------------------
758
The first thing to do is to dump them into a format that facilitates later processing with Windows PowerShell.
759
 
760
To dump the event log, you can use the Get-EventLog and the Exportto-Clixml cmdlets if you are working with a traditional event log such as the Security, Application, or System event logs.
761
If you need to work with one of the trace logs, use the Get-WinEvent and the ExportTo-Clixml cmdlets.
762
 
763
Get-EventLog -LogName application | Export-Clixml Applog.xml
764
 
765
type .\Applog.xml
766
 
767
$logs = "system","application","security"
768
 
769
The % symbol is an alias for the Foreach-Object cmdlet. It is often used when working interactively from the Windows PowerShell console
770
 
771
$logs | % { get-eventlog -LogName $_ | Export-Clixml "$_.xml" }
772
 
773
 
774
 
775
Step 2: Import the event log of interest
776
----------------------------------------
777
To parse the event logs, use the Import-Clixml cmdlet to read the stored XML files.
778
Store the results in a variable.
779
Let's take a look at the commandlets Where-Object, Group-Object, and Select-Object.
780
 
781
The following two commands first read the exported security log contents into a variable named $seclog, and then the five oldest entries are obtained.
782
 
783
$seclog = Import-Clixml security.xml
784
 
785
$seclog | select -Last 5
786
 
787
 
788
Cool trick from one of our students named Adam. This command allows you to look at the logs for the last 24 hours:
789
 
790
Get-EventLog Application -After (Get-Date).AddDays(-1)
791
 
792
You can use '-after' and '-before' to filter date ranges
793
 
794
One thing you must keep in mind is that once you export the security log to XML, it is no longer protected by anything more than the NFTS and share permissions that are assigned to the location where you store everything.
795
By default, an ordinary user does not have permission to read the security log.
796
 
797
 
798
Step 3: Drill into a specific entry
799
-----------------------------------
800
To view the entire contents of a specific event log entry, choose that entry, send the results to the Format-List cmdlet, and choose all of the properties.
801
 
802
 
803
$seclog | select -first 1 | fl *
804
 
805
The message property contains the SID, account name, user domain, and privileges that are assigned for the new login.
806
 
807
 
808
($seclog | select -first 1).message
809
 
810
(($seclog | select -first 1).message).gettype()
811
 
812
 
813
 
814
In the *nix world you often want a count of something (wc -l).
815
How often is the SeSecurityPrivilege privilege mentioned in the message property?
816
To obtain this information, pipe the contents of the security log to a Where-Object to filter the events, and then send the results to the Measure-Object cmdlet to determine the number of events:
817
 
818
$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | measure
819
 
820
If you want to ensure that only event log entries return that contain SeSecurityPrivilege in their text, use Group-Object to gather the matches by the EventID property.
821
 
822
 
823
$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | group eventid
824
 
825
Because importing the event log into a variable from the stored XML results in a collection of event log entries, it means that the count property is also present.
826
Use the count property to determine the total number of entries in the event log.
827
 
828
$seclog.Count
829
 
830
 
831
 
832
 
833
 
834
 
835
############################
836
# Simple Log File Analysis #
837
############################
838
 
839
 
840
You'll need to create the directory c:\ps and download sample iss log http://pastebin.com/raw.php?i=LBn64cyA
841
 
842
 
843
mkdir c:\ps
844
cd c:\ps
845
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
846
 
847
 
848
 
849
 
850
       
851
       
852
       
853
 
854
###############################################
855
# Intrusion Analysis Using Windows PowerShell #
856
###############################################
857
 
858
Download sample file http://pastebin.com/raw.php?i=ysnhXxTV into the c:\ps directory
859
 
860
 
861
 
862
 
863
 
864
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=ysnhXxTV", "c:\ps\CiscoLogFileExamples.txt")
865
 
866
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt
867
 
868
 
869
 
870
 
871
The Select-String cmdlet searches for text and text patterns in input strings and files. You can use it like Grep in UNIX and Findstr in Windows.
872
 
873
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line
874
 
875
 
876
 
877
 
878
To see how many connections are made when analyzing a single host, the output from that can be piped to another command: Measure-Object.
879
 
880
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line | Measure-Object
881
 
882
 
883
 
884
To select all IP addresses in the file expand the matches property, select the value, get unique values and measure the output.
885
 
886
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique | Measure-Object
887
 
888
 
889
 
890
Removing Measure-Object shows all the individual IPs instead of just the count of the IP addresses. The Measure-Object command counts the IP addresses.
891
 
892
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique
893
 
894
 
895
In order to determine which IP addresses have the most communication the last commands are removed to determine the value of the matches. Then the group command is issued on the piped output to group all the IP addresses (value), and then sort the objects by using the alias for Sort-Object: sort count –des.
896
This sorts the IP addresses in a descending pattern as well as count and deliver the output to the shell.
897
 
898
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select value | group value | sort count -des
899
 
900
 
901
 
902
 
903
This will get the setting for logs in the windows firewall which should be enabled in GPO policy for analysis.
904
The command shows that the Firewall log is at:
905
%systemroot%\system32\LogFiles\Firewall\pfirewall.log, in order to open the file PowerShell will need to be run with administrative privileges.
906
 
907
 
908
First step is to get the above command into a variable using script logic.
909
Thankfully PowerShell has a built-in integrated scripting environment, PowerShell.ise.
910
 
911
netsh advfirewall show allprofiles | Select-String FileName | select -ExpandProperty line | Select-String “%systemroot%.+\.log" | select -ExpandProperty matches | select -ExpandProperty value | sort –uniq
912
 
913
 
914
##############################################
915
# Parsing Log files using windows PowerShell #
916
##############################################
917
 
918
Download the sample IIS log http://pastebin.com/LBn64cyA
919
 
920
 
921
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
922
 
923
Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV")}  
924
 
925
 
926
 
927
The above command would give us all the WebDAV requests.
928
 
929
To filter this to a particular user name, use the below command:
930
 
931
Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "OPTIONS")}  
932
 
933
 
934
 
935
Some more options that will be more commonly required :
936
 
937
For Outlook Web Access : Replace WebDAV with OWA
938
 
939
For EAS : Replace WebDAV with Microsoft-server-activesync
940
 
941
For ECP : Replace WebDAV with ECP
942
 
943
 
944
 
945
To find out the count of the EWS request we can go ahead and run the below command
946
 
947
(Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "Useralias")}).count
948
949
950
951
952
953
954
955
956
 
957
Explain to me how this script works.