Querying DSC File Sets

This section describes some of the most common ways to use LINQ to HPC to query data that resides in a DSC file set. See Creating DSC File Sets for information about how to create DSC file sets from existing data. If you want to learn about more complex queries, see Implementing Distributed Algorithms by Using LINQ to HPC.

 

Scenario

To query the DSC file set data…

You have a DSC file set that is made up of lines of text. For example, you want to analyze web server log files.

Use the FromDsc method, and use LineRecord as the type parameter. For more information, see Using the LineRecord Class to Query a Text-Based File Set.

You have a DSC file set that is in binary format. It contains serialized .NET objects. For example, you have adapted an existing data parallel .NET program to cluster computing.

Use the FromDsc method, and use a type parameter that matches the type that was originally used to save the data. For more information, see Querying a Binary DSC File Set.

You want to extract the results of a LINQ to HPC query into a locally running application.

Enumerate the values of the query. For more information, see Enumerating the Results of a LINQ to HPC Query.

You want to transform the data in a DSC file set, and then save the results as a new file set.

Use the FromDsc method to read the input file set. Use the Select operator to process each item. Finally, use the ToDsc method to save the results as a new file set. See Saving the Results of a LINQ to HPC Query to a New DSC File Set.

You can use almost any LINQ operator in a LINQ to HPC query. The following table lists the supported query operators.

 

Operator Type

Operator

Restriction Operators

Where

Projection Operators

Select, SelectMany

Partitioning Operators

Take, Skip, TakeWhile, SkipWhile

Join Operators

Join, GroupJoin

Ordering Operators

OrderBy, OrderByDescending, Reverse

Grouping Operators

GroupBy

Set Operators

Concat, Distinct, Union, Intersect, Except

Conversion Operators

AsEnumerable, ToArray, ToList, ToDictionary, ToLookup, Cast

Element Operators

First, FirstOrDefault, Last, LastOrDefault, Single, SingleOrDefault, DefaultIfEmpty

Aggregate Operators

Count, LongCount, Sum, Max, Min, Average, Aggregate, GroupBy

Quantifiers

All, Any, Contains

noteNote
The following LINQ operators cannot be used in LINQ to HPC queries: ElementAt, ElementAtOrDefault, OfType, SequenceEqual, ThenBy, ThenByDescending.

Unlike LINQ, LINQ to HPC allows delayed execution for scalar queries. LINQ to HPC supports the following operators that return queries instead of immediate results. See Creating Free-Form Compound Queries.

  • AggregateAsQuery

  • AllAsQuery

  • AnyAsQuery

  • AverageAsQuery

  • ContainsAsQuery

  • CountAsQuery

  • FirstAsQuery

  • FirstOrDefaultAsQuery

  • LastAsQuery

  • LastOrDefaultAsQuery

  • LongCountAsQuery

  • MaxAsQuery

  • MinAsQuery

  • SingleAsQuery

  • SingleOrDefaultAsQuery

  • SumAsQuery



Show: