Creating DSC File Sets for Intermediate Data

When you run a query that contains multiple stages, the LINQ to HPC runtime creates and manages file sets that store the intermediate results of the queries.

You can use the IntermediateDataCompressionScheme property to control whether intermediate results should be stored in a compressed format to conserve space, or whether the results should be uncompressed to conserve processing resources. The default setting is to use compression. By default, intermediate data is kept for one hour for successful jobs, and twenty-four hours for failed jobs.



Show: