U-SQL Language Reference

 

Published: October 28, 2015

Updated: March 10, 2017

Introduction

This is the reference documentation for the U-SQL language.

What is U-SQL?

U-SQL is the new big data query language of the Azure Data Lake Analytics service.

It evolved out of Microsoft's internal Big Data language called SCOPE and combines a familiar SQL-like declarative language with the extensibility and programmability provided by C# types and the C# expression language and big data processing concepts such as “schema on reads”, custom processors and reducers. It also provides the ability to query and combine data from a variety of data sources, including Azure Data Lake Storage, Azure Blob Storage, and Azure SQL DB, Azure SQL Data Warehouse, and SQL Server instances running in Azure VMs.

It is however not ANSI SQL. For starters, its keywords such as SELECT have to be in UPPERCASE. And its expression language inside SELECT clauses, WHERE predicates etc is C#. This for example means, that the comparison operations inside a predicate follow C# syntax (e.g., a == "foo"), and that the language uses C# null semantics which is 2-valued and not 3-valued as in ANSI SQL. To help SQL users to get familiar with U-SQL, a section is providing the mapping of some common SQL expressions and how to express them in U-SQL.

Overview of the Language Reference

The language reference is organized as follows:

  1. This introduction provides a quick introduction and overview of the reference documentation.

  2. Introduction of the U-SQL concepts
    a. U-SQL Processing Model
    b. U-SQL Script Structure
    c. Files as inputs and outputs
    d. Rowset: The Processing Data Structure
    e. U-SQL and Identifiers
    f. U-SQL Expressions
    g. Metadata system and objects (Content is under development and will be provided soon)
    h. Language extensibility (Content is under development and will be provided soon)

  3. Syntax and Semantics of the U-SQL Language
    a. Types and Literals (U-SQL)
    b. Data Definition Language (DDL) Statements (U-SQL)
    c. U-SQL Query Language
    d. Data Modification Language (DML) Statements (U-SQL)
    e. Outputting to files with U-SQL
    f. Built-in Functions (U-SQL)

  4. How to express common SQL expressions in U-SQL

If you are interested in seeing how you use the above concepts to solve certain scenarios please refer to the Language User Guide, tutorials as well as the U-SQL Programmability Guide.

See Also

  • In order to see how the above concepts can be applied to solve certain scenarios, please refer to the following:
  • For additional information please refer to:
    • U-SQL C# Developer’s Guide (under development)
    • U-SQL Best Practices and Optimizations (under development)

Community Additions

ADD
Show: