How to make Sub faster/more efficient? --Takes to long

Forums

My Profile

Inbox

Address Book

My Subscription

My Forums

How to make Sub faster/more efficient? --Takes to long

Logged in as: Guest

arrSession:exec spGetSession 2,2,61950

	Active Users: There are 0 members and 0 guests.
	Users viewing this topic: none

Printable Version

All Forums >> [Scripting] >> WSH & Client Side VBScript >> How to make Sub faster/more efficient? --Takes to long
Do you like VisualBasicScript.com? Link to us and help spread the word about our forum. Thanks!

Page: [1]

Message

<< Older Topic Newer Topic >>

How to make Sub faster/more efficient? --Takes to long - 6/28/2008 7:34:39 AM

yfki

Posts: 73
Score: 0
Joined: 12/12/2007
Status: offline

This sub runs on a few .txt files, each in range from 25-125MB
When the sub processes the larger logs, seems to take much longer then it should.

How can I make this sub run fater?

Sub ParseLogFiles()

Dim objFSO, WshShell, objRegEx, colMatches, cTempDir
   Dim strMonth, strDay, objTextFile, strNextLine, arrLogEntries, LogEntry, strLogFile
   Dim strPattern, strMatch, strYesterday
   Dim objFolder, colFiles
    
   Const ForReading = 1
   Set objFSO = CreateObject("Scripting.FileSystemObject")
   Set WshShell = CreateObject("Wscript.Shell")
   Set objRegEx = CreateObject("VBScript.RegExp")
       objRegEx.IgnoreCase = True
           
   strYesterday = Date() -1
   objRegEx.Pattern = "([1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](20\d\d)"
       
   Set colMatches = objRegex.Execute(strYesterday)
           strMonth = colMatches(0).Submatches(0)
           strDay = colMatches(0).Submatches(1)
           strYear = colMatches(0).Submatches(2)
   
   If Day(Now) < 10 Then 
       strDay =  "0" & strDay
   End If

If Month(Now) < 10 Then 
       strMonth =  "0" & strMonth
   End If

Set objFolder = objFSO.GetFolder(strTempDir)
   Set colFiles = objFolder.Files
       For Each strLogFile In colFiles  
       Set objTextFile = objFSO.OpenTextFile(strLogFile, ForReading)
   
       Do Until objTextFile.AtEndOfStream
           strNextLine = objTextFile.Readall
           arrLogEntries = Split(strNextLine , vbcrlf)
       Loop

For Each LogEntry In arrLogEntries
       On Error Resume Next
           strPattern = strYear & "-" & strMonth & "-" & strDay
           objRegEx.Pattern = strPattern
           If objRegEx.Test(LogEntry) = True Then 
               objRegEx.Pattern = "(RESULT=""Success"")"
               If objRegEx.Test(LogEntry) = True Then 
                   objRegEx.Pattern = "(\\\\[\w-]+\\[^\\/:*?<>|""]+)((?:\\[^\\/:*?<>|""]+)*\\?)"
                   Set strMatch = objRegEx.Execute(LogEntry)
                    Call LogToFile(strMatch(0).value, "Migrations_" & strYear & strMonth & strDay & ".txt")
                       strMigratedFileCount = strMigratedFileCount + 1
                       Set strMatch = Nothing
                   If Err.Number <> 0 Then 
                    Call LogToFile("An error has occurred matching the UNC path of this log entry, it " & _
                                          "seems to be a directory, not a file.  " & VbCrLf & "Log Entry: " & LogEntry, "ReportErrors_" & strYear & strMonth & strDay & ".txt")                    
                   End If
               End If
           End If
         On Error GoTo 0
       Next
   Next
End Sub

Post #: 1

RE: How to make Sub faster/more efficient? --Takes to long - 6/28/2008 8:02:31 AM

dm_4ever

Posts: 2433
Score: 38
Joined: 6/29/2006
From: Orange County, California
Status: offline

Just a little something to add...

You don't necessarily need a RegExp for the date...the following would work

strMonth = Right("00" & Month(strYesterday), 2)
strDay = Right("00" & Day(strYesterday), 2)
strYear = Year(strYesterday)

If you're going to read the entire content...you don't need to use the Do Until.... used below.

     Do Until objTextFile.AtEndOfStream
         strNextLine = objTextFile.Readall
         arrLogEntries = Split(strNextLine , vbcrlf)
     Loop

...you could simply do........

arrLogEntries = Split(objTextFile.ReadAll, vbCrLf)

However, since your intent is to go through each line...then why bother adding it to an array...why not

    Do Until objTextFile.AtEndOfStream
        strLine = objTextFile.ReadLine

        strPattern = strYear & "-" & strMonth & "-" & strDay
        objRegEx.Pattern = strPattern
        If objRegEx.Test(strLine) Then
            objRegEx.Pattern = "(RESULT=""Success"")"
            If objRegEx.Test(strLine) Then
                objRegEx.Pattern = "(\\\\[\w-]+\\[^\\/:*?<>|""]+)((?:\\[^\\/:*?<>|""]+)*\\?)"
                If objRegEx.Test(strLine)
                    Call LogToFile(objRegEx.Execute(strLine).SubMatches(0), "Migrations_" & strYear & strMonth & strDay & ".txt")
                    strMigratedFileCount = strMigratedFileCount + 1
                Else
                    Call LogToFile("An error has occurred matching the UNC path of this log entry, it " & _
                    "seems to be a directory, not a file. " & VbCrLf & "Log Entry: " & strLine, "ReportErrors_" & strYear & strMonth & strDay & ".txt")
                End If
            End If
        End If
    Loop

      objTextFile.Close

....oh and you don't have to say objRegEx.Test(xxx) = True since this is a bit redundant

< Message edited by dm_4ever -- 6/28/2008 8:03:45 AM >

_____________________________

dm_4ever

My philosophy: K.I.S.S - Keep It Simple Stupid
Read Me: http://www.visualbasicscript.com/m_24727/tm.htm
Frequently Asked Stuff: http://www.visualbasicscript.com/m_47117/tm.htm

(in reply to yfki)

Post #: 2

RE: How to make Sub faster/more efficient? --Takes to long - 6/28/2008 1:20:48 PM

yfki

Posts: 73
Score: 0
Joined: 12/12/2007
Status: offline

DM

Thx a ton, I shaved a ton of time off.
Took the same approach all over the place.

In general, is it better to process things within less sub-routine or function?
Is performance degraded at all by the time it takes to pass thigns back and forth when you modularize things?

I always have a tendancy to use a sub to do a couple things, then pass it off to something else for continues processing.
Never really had to think about how fast the stuff is processed since it wa alwyas very lightweight data.

Parsing these 100mb logs make for interesting performance tweaking.

eitherway, DM, thx a ton!

(in reply to dm_4ever)

Post #: 3

RE: How to make Sub faster/more efficient? --Takes to long - 6/28/2008 4:39:48 PM

dm_4ever

Posts: 2433
Score: 38
Joined: 6/29/2006
From: Orange County, California
Status: offline

I tend to break just about everything I do into a sub or function and try to stay away from global variables as much as possible....if a sub/functions will be called several times and it creates a particular object several times...say like WScript.Shell or Scripting.FileSystem then I will normally create it once and pass it to any other sub or function (that's my personal prefrence and I don't recall what I did, but it did make a slight difference; though in most cases having it be self contained might be worth sacrificing a few seconds)

One other thing you may consider with parsing such large files is seeing if you can find a RegEx pattern that will match exactly what you're looking for in one shot.

Set colMatches = objRegEx.Execute(objTextFile.ReadAll)
For Each objMatch in colMatches
...code that deals with each match
Next

_____________________________

dm_4ever

My philosophy: K.I.S.S - Keep It Simple Stupid
Read Me: http://www.visualbasicscript.com/m_24727/tm.htm
Frequently Asked Stuff: http://www.visualbasicscript.com/m_47117/tm.htm

(in reply to yfki)

Post #: 4

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 12:52:31 AM

DiGiTAL.SkReAM

Posts: 1157
Score: 7
Joined: 9/6/2005
From: Florida, USA
Status: offline

dm_4ever:
>...you could simply do........
>
>arrLogEntries = Split(objTextFile.ReadAll, vbCrLf)
>
>However, since your intent is to go through each line...then why bother adding it to an array...why not

http://www.visualbasicscript.com/fb.aspx?m=47020 shows some examples of performance issues with reading large files.
At the filesizes the OP is using, it might be faster to read the entire file than to process it in a line-by-line fashion.

Also I suspect that a large amount of the processing 'overhead' is having to do with the RegEx usage. When comparing InStr() vs. RegEx, InStr() will win every time - as long as it is a case sensitive search.
RegEx is useful/nice when using it against a few things, but when using it against a file that might have thousands upon thousands of lines.... it becomes quite the anchor around your feet.

_____________________________

"Would you like to touch my monkey?" - Dieter (Mike Meyers)

"It is better to die like a tiger, than to live like a pussy."
-Master Wong, from Balls of Fury

(in reply to dm_4ever)

Post #: 5

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 2:09:51 AM

yfki

Posts: 73
Score: 0
Joined: 12/12/2007
Status: offline

So revamped the script, removed 4 subs, and combined a bunch of other stuff.

Before I was getting the files for pasring, then parsing the files entries for the UNC path, writing to a file, then reading the file into an array, and retrieving the size of each file.
and finally writing that to a new file etc....(This approach was originally done, in fear of memory issues, did'nt want to load that much stuff)

There was way to much overhead.
With removing the extra subs, and updating the Parse() sub to handle alot more in the For Each loop, I dont have to load everything into memeory anymore.
Including DM's reccomendations, speed has improoved dramatically!
Look at the memory consumption as well.

Some results:

Before:
--------------------------
HSM Migration Report
6/29/2008 2:22:58 AM
--------------------------
Total Files:    87896
Total Size:    521 GB
                  533,743 MB
Run Time: 68 minutes
Working Set Size Peak: 368,557 K

After:
--------------------------
HSM Migration Report
6/29/2008 5:22:58 AM
--------------------------
Total Files:    87896
Total Size:    521 GB
                  533,743 MB
Run Time: 21 minutes
Working Set Size Peak: 4096 K

(in reply to DiGiTAL.SkReAM)

Post #: 6

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 4:42:00 AM

dm_4ever

Posts: 2433
Score: 38
Joined: 6/29/2006
From: Orange County, California
Status: offline

InStr() is faster, but it also has its limitations...especially if you intend on extracting specific pieces of information or are looking for patterns rather than specific values.

If anything I think using the InStr() function to check for the date and the "Result = "Success"" would provide some additional speed improvements...but the RegExp would still be my choice to extract the path...it's simply easier than having to do InStr() with Mid/Left/Right functions to extract that info.

This has been discussed several times and I think it really depends on what you ultimate goal is...it sometimes makes a difference.
http://www.visualbasicscript.com/m_44624/tm.htm
http://www.visualbasicscript.com/m_43401/tm.htm

< Message edited by dm_4ever -- 6/29/2008 4:46:39 AM >

_____________________________

dm_4ever

My philosophy: K.I.S.S - Keep It Simple Stupid
Read Me: http://www.visualbasicscript.com/m_24727/tm.htm
Frequently Asked Stuff: http://www.visualbasicscript.com/m_47117/tm.htm

(in reply to DiGiTAL.SkReAM)

Post #: 7

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 5:30:41 AM

TNO

Posts: 1066
Score: 10
Joined: 12/18/2004
From: thenewobjective.com
Status: offline

Could you post your new sub? After the obvious optimizations are done, some not so obvious ones (Like loop unrolling) may help

_____________________________

To iterate is human, to recurse divine. -- L. Peter Deutsch

(in reply to dm_4ever)

Post #: 8

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 5:55:51 AM

yfki

Posts: 73
Score: 0
Joined: 12/12/2007
Status: offline

Here is the whole thing, there is still plenty to clean up. But works like a charm right now.
I am only avg vbscript dev, so I'm open to any and all suggestions.

The 2 regex checks in the ParseLogFiles I will probably change to Instr and see what that does.
This script is used to process Tivoli HSM migration logs, calcualte total number of files migrated,
and calcualte the total size of all migrations.

Option Explicit

Dim strDay, strMonth, strYear, strTotalSize, strYesterday 
Dim bEnableLogging, bIncludeDateStamp, bPrependDateStampInLogFileName
Dim sLogFileLocation, sOverWriteORAppend, strTempDir
Dim strMigratedFileTotalSize, strMigratedFileCount, strHSMLogs, strNowStart
Dim objRegEx4Date, strDate

strNowStart = Now
strTempDir = ""
strHSMLogs = ""
bEnableLogging = True
bIncludeDateStamp = False
bPrependDateStampInLogFileName = False
sLogFileLocation = "relative"
sOverWriteORAppend = "append"
strMigratedFileTotalSize = 0
strMigratedFileCount = 0
   
Set objRegEx4Date = CreateObject("VBScript.RegExp")
   objRegEx4Date.IgnoreCase = True
           
   strYesterday = Date() -1
   objRegEx4Date.Pattern = "([1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](20\d\d)"
   
   Set strDate  = objRegEx4Date.Execute(strYesterday)
   strMonth = strDate(0).Submatches(0)
   strDay   = strDate(0).Submatches(1)
   strYear  = strDate(0).Submatches(2)
   
   If Day(Now) < 10 Then 
       strDay =  "0" & strDay
   End If

If Month(Now) < 10 Then 
       strMonth =  "0" & strMonth
   End If

Call CleanTemp
Call GetFileForParsing
Call ParseLogFiles
Call GenerateReport
Call SendReport
Call CleanTemp

'=======================================================================================================
'=======================================================================================================

Sub GetFileForParsing
   Dim objFSO, objRegEx, cTempDir
   Dim objStartFolder, objFolder, colFiles, objFile, strDateDifference, LogsToKeep, colMatches 
   Dim arrLogsToParse, strLogFileToParse, strOnlyLogFileName 
   
   Set objRegEx = CreateObject("VBScript.RegExp")
           objRegEx.IgnoreCase = True
           objRegEx.Pattern = "migrate"
   Set objFSO = CreateObject("Scripting.FileSystemObject")
           objStartFolder = strHSMLogs 
   
       
   Set objFolder = objFSO.GetFolder(objStartFolder)
       'Wscript.Echo objFolder.Path
   
   Set colFiles = objFolder.Files
   
   For Each objFile In colFiles
       If objRegEx.Test(objFile.name) Then 
         strDateDifference = DateDiff("h", objFile.DateLastModified, Now)
             If strDateDifference < 48 Then            
                 LogsToKeep = LogsToKeep & objFile.Path & ","
           End If            
       End If
   Next

LogsToKeep = Left(LogsToKeep,Len(LogsToKeep)-1)
   ArrLogsToParse = Split(LogsToKeep, ",")
   
       For Each strLogFileToParse In arrLogsToParse 
           objRegEx.Pattern = "(\\.+\\)(.+)"
       Set colMatches = objRegEx.Execute(strLogFileToParse)
           strOnlyLogFileName = colMatches(0).SubMatches(1)
           objFSO.CopyFile strLogFileToParse, strTempDir & strOnlyLogFileName, True
       Next
End Sub

Sub ParseLogFiles()

Dim objFSO, WshShell, objRegEx, colMatches, cTempDir
   Dim objTextFile, strNextLine, arrLogEntries, LogEntry, strLogFile, strLine
   Dim strPattern, strMatch, strYesterday
   Dim objFolder, objFile, colFiles, strMigratedFile
    
   Const ForReading = 1
   Set objFSO = CreateObject("Scripting.FileSystemObject")
   Set WshShell = CreateObject("Wscript.Shell")
   Set objRegEx = CreateObject("VBScript.RegExp")
       objRegEx.IgnoreCase = True
           
   
   Set objFolder = objFSO.GetFolder(strTempDir)
   Set colFiles = objFolder.Files
       For Each strLogFile In colFiles  
           Set objTextFile = objFSO.OpenTextFile(strLogFile, ForReading)
       
   Do Until objTextFile.AtEndOfStream
         strLine = objTextFile.ReadLine       
         objRegEx.Pattern = strYear & "-" & strMonth & "-" & strDay
         If objRegEx.Test(strLine) Then
             objRegEx.Pattern = "(RESULT=""Success"")"
             If objRegEx.Test(strLine) Then
                 objRegEx.Pattern = "(\\\\[\w-]+\\[^\\/:*?<>|""]+)((?:\\[^\\/:*?<>|""]+)*\\?)"
                 If objRegEx.Test(strLine) Then
           On Error Resume Next
           Set strMatch = objRegEx.Execute(strLine)
                     Set objFile = objFSO.GetFile(strMatch(0).value)
                   Call LogToFile(objFile.Size & "," & strMatch(0).value, "hsm_MigrationSize_" & strYear & strMonth & strDay & ".txt")        
                       strMigratedFileTotalSize = strMigratedFileTotalSize + objFile.Size 
                   Set strMatch = Nothing
                         strMigratedFileCount = strMigratedFileCount + 1
                       Else
                       Call LogToFile("An error has occurred matching the UNC path of this log entry, it " & _
                                  "seems to be a directory, not a file.  " & VbCrLf & "Log Entry: " & strLine, _
                        "ReportErrors_" & strYear & strMonth & strDay & ".txt")                   
           On Error GoTo 0
                 End If
             End If
         End If
       Loop
     objTextFile.Close 
  Next
End Sub

Sub GenerateReport
Dim strMigratedFileTotalSizeMB, strMigratedFileTotalSizeGB
   strMigratedFileTotalSizeMB = FormatNumber(((strMigratedFileTotalSize/1024)/1024),0) & " MB"     
   strMigratedFileTotalSizeGB = FormatNumber((((strMigratedFileTotalSize/1024)/1024)/1024),0) & " GB" 
   LogToFile "--------------------------" & VbCrLf & _
         "HSM Migration Report" & VbCrLf & Date() -1 & VbCrLf & _
         "--------------------------" & VbCrLf & _
         "Total Files:" & vbTab & strMigratedFileCount & VbCrLf & _
         "Total Size:" & vbTab & strMigratedFileTotalSizeGB & VbCrLf & _
         vbTab & vbTab & strMigratedFileTotalSizeMB & VbCrLf & _
         "Run Time: " & DateDiff("n", strNowStart, Now) & " minutes", _
         "hsm_MigrationReport_" & strYear & strMonth & strDay & ".txt"
End Sub

Sub SendReport

Dim WshShell, objEmail
   
   Set WshShell = CreateObject("Wscript.Shell")
   Set objEmail = CreateObject("CDO.Message")
   objEmail.From = ""
   objEmail.To = ""
   objEmail.Subject = "HSM Migration - Daily Report " & Date()
   'msgbox(WshShell.CurrentDirectory & "\hsm_MigrationReport_" & strYear & strMonth & strDay & ".txt")
   objEmail.AddAttachment(WshShell.CurrentDirectory & "\hsm_MigrationReport_" & strYear & strMonth & strDay & ".txt")

objEmail.Configuration.Fields.Item _
       ("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
   objEmail.Configuration.Fields.Item _
       ("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "" '
   objEmail.Configuration.Fields.Item _
       ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
   objEmail.Configuration.Fields.Update
   objEmail.Send
   Set objEmail = Nothing

End Sub

Sub CleanTemp
   Dim objFSO
   Set objFSO = CreateObject("Scripting.FileSystemObject")
   objFSO.DeleteFile(strTempDir & "*.*"), True
End Sub

Sub LogToFile(Message, sLogFileName)

Dim oLogFSO, oLogShell, sNow, oLogFile, sLogFile
       
   If bEnableLogging = False Then Exit Sub

Const ForWriting = 2
   Const ForAppending = 8

Set oLogFSO = CreateObject("Scripting.FileSystemObject")
  
   If sLogFileLocation = "relative" Then
       Set oLogShell = CreateObject("Wscript.Shell")
       sLogFileLocation = oLogShell.CurrentDirectory & "\"
       Set oLogShell = Nothing
   End If
  
   If bPrependDateStampInLogFileName Then
       sNow = Replace(Replace(Now(),"/","-"),":",".")
       sLogFileName = sNow & " - " & sLogFileName
       bPrependDateStampInLogFileName = False       
   End If
  
   sLogFile = sLogFileLocation & sLogFileName
  
   If sOverWriteORAppend = "overwrite" Then
       Set oLogFile = oLogFSO.OpenTextFile(sLogFile, ForWriting, True)
       sOverWriteORAppend = "append"
   Else
       Set oLogFile = oLogFSO.OpenTextFile(sLogFile, ForAppending, True)
   End If

If bIncludeDateStamp Then
       Message = Now & "   " & Message
   End If

oLogFile.WriteLine(Message)
  
   oLogFile.Close
   oLogFSO = Null
End Sub

(in reply to TNO)

Post #: 9

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 6:49:14 AM

dm_4ever

Posts: 2433
Score: 38
Joined: 6/29/2006
From: Orange County, California
Status: offline

In the first sub you may be able to combine the last to For Each... into one

i.e.

   For Each objFile In colFiles
       If InStr(LCase(objFile.Name), "migrate") > 0 Then
           strDateDifference = DateDiff("h", objFile.DateLastModified, Now)
           If strDateDifference < 48 Then
               objFile.Copy strTempDir & objFile.Name, True
           End If
       End If
   Next

_____________________________

dm_4ever

My philosophy: K.I.S.S - Keep It Simple Stupid
Read Me: http://www.visualbasicscript.com/m_24727/tm.htm
Frequently Asked Stuff: http://www.visualbasicscript.com/m_47117/tm.htm

(in reply to yfki)

Post #: 10

RE: How to make Sub faster/more efficient? --Takes to long - 6/29/2008 7:15:12 AM

yfki

Posts: 73
Score: 0
Joined: 12/12/2007
Status: offline

Reduced to...

(in reply to dm_4ever)

Post #: 11

If you found our site useful please link to us <a href="http://www.visualbasicscript.com">VisualBasicScript.com</a>.

All Forums >> [Scripting] >> WSH & Client Side VBScript >> How to make Sub faster/more efficient? --Takes to long

Page: [1]

Jump to:

New Messages	No New Messages
Hot Topic w/ New Messages	Hot Topic w/o New Messages
Locked w/ New Messages	Locked w/o New Messages

Post New Thread

Reply to Message

Post New Poll

Submit Vote

Delete My Own Post

Delete My Own Thread

Rate Posts