How to: Programmatically Export the Crawl History to a CSV File [Search Server 2008]
Microsoft Search Server 2008 tracks crawl statistics, including the following:
-
Crawl type (full or incremental)
-
Crawl start time and end time
-
Content source associated with the crawl
-
Number of items crawled successfully, and the number of warnings and errors that occurred
You can use the Search Administration object model to retrieve this information by using the Microsoft.Office.Server.Search.Administration.CrawlHistory class. This class contains the GetCrawlHistory function, which returns the crawl statistics in a DataTable object.
The following procedure shows how to write out the contents of the crawl history table to a comma separated values (.csv) file. You can then open that file in an application such as Microsoft Office Excel for further analysis.
To export the crawl history to a .csv file from a console application
-
In Microsoft Visual Studio, on the File menu, point to New, and then click Project.
-
In Project types, under C#, click Windows.
-
Under Templates, click Console Application. In the Name field, type CrawlHistorySample, and then click OK.
-
On the Project menu, click Add Reference.
-
On the .NET tab, select the following references, and then click OK:
-
Microsoft Search component (Microsoft.Office.Server.Search.dll)
-
Microsoft Office Server component (Microsoft.Office.Server.dll)
-
Windows SharePoint Services (Microsoft.SharePoint.dll)
-
-
In your console application's class file, add the following using statements near the top of the code with the other namespace directives.
using System.IO; using System.Data; using Microsoft.SharePoint; using Microsoft.Office.Server.Search.Administration;
-
Create a function to write out usage information to the console window.
static void Usage() { Console.WriteLine("Export Crawl History to CSV File Sample"); Console.WriteLine("Usage: CrawlHistorySample.exe SiteURL OutputPath OutputName"); Console.WriteLine(@"Example: CrawlHistorySample.exe http://MySearchServer c:\logfiles\ CrawlHistoryLog1"); }
-
In the Main function of the console application, add the following code to open a try block.
try { -
Add code to check the number of items in the args[] parameter; this number must be equal to 3. If it does not, then call the Usage function defined in step 7.
if (args.Length != 3) { Usage(); return; }
-
Retrieve the values specified in the args[] parameter, to be used to specify the search center site URL, the output path, and name for the .csv file.
string strURL = args[0]; string csvPath = args[1] + "\\" + args[2] + ".csv";
-
Add the following code to retrieve the SearchContext.
SearchContext context; using (SPSite site = new SPSite(strURL)) { context = SearchContext.GetContext(site); }
-
Retrieve the crawl history into a System.Data.DataTable object by using the following code.
CrawlHistory history = new CrawlHistory(context); DataTable table = new DataTable(); table = history.GetCrawlHistory();
-
Add the following code to initialize the System.IO.StreamWriter object.
using (StreamWriter writer = new StreamWriter(csvPath)) { /* Code to iterate through DataTable rows, and write out the values to the .csv file See steps 14 through 17. */
-
Retrieve the number of columns, as follows.
int colCount = table.Columns.Count; -
Write out the column names to the file stream, with a comma following each name, except for the last column name, and then write a new line to the file stream, as follows.
for (int i = 0; i < colCount; i++) { writer.Write(table.Columns[i].ToString()); if (i < colCount - 1) { writer.Write(","); } } writer.Write(writer.NewLine);
-
Loop through the collection of rows, and write out each column value to the file stream, with a comma following each value. Write out a new line to the file stream at the end of the row before moving to the next row.
foreach (DataRow row in table.Rows) { for (int i = 0; i < colCount; i++) { writer.Write(row[i].ToString()); if (i < colCount - 1) { writer.Write(","); } } writer.Write(writer.NewLine); }
-
Flush the contents of the file stream, close it, and then close the using statement block and the try block, as follows.
writer.Flush(); writer.Close(); } } -
Add the following code for the catch block.
catch (Exception ex) { Console.WriteLine(ex.ToString()); Usage(); }
Example
Following is the complete code for the CrawlHistorySample console application class.
using System; using System.Collections.Generic; using System.Text; using System.IO; using System.Data; using Microsoft.Office.Server.Search.Administration; using Microsoft.SharePoint; namespace CrawlHistorySample { class Program { static void Main(string[] args) { try { if (args.Length != 3) { Usage(); return; } string strURL = args[0]; string csvPath = args[1] + "\\" + args[2] + ".csv"; SearchContext context; using (SPSite site = new SPSite(strURL)) { context = SearchContext.GetContext(site); } CrawlHistory history = new CrawlHistory(context); DataTable table = new DataTable(); table = history.GetCrawlHistory(); using (StreamWriter writer = new StreamWriter(csvPath, false)) { int colCount = table.Columns.Count; for (int i = 0; i < colCount; i++) { writer.Write(table.Columns[i].ToString()); if (i < colCount - 1) { writer.Write(","); } } writer.Write(writer.NewLine); foreach (DataRow row in table.Rows) { for (int i = 0; i < colCount; i++) { writer.Write(row[i].ToString()); if (i < colCount - 1) { writer.Write(","); } } writer.Write(writer.NewLine); } writer.Flush(); writer.Close(); } } catch (Exception ex) { Console.WriteLine(ex.ToString()); Usage(); } } static void Usage() { Console.WriteLine("Export Crawl History to CSV File Sample"); Console.WriteLine("Usage: CrawlHistorySample.exe SiteURL OutputPath OutputName"); Console.WriteLine(@"Example: CrawlHistorySample.exe http://MySearchServer c:\logfiles\ CrawlHistoryLog1"); } } }