{ End Bracket }: The World-Wide Telescope

MSDN Magazine

Printer Friendly Version

Send

MSDN

{ End Bracket }: The World-Wide Tel...

Toolbox: Static Analysis Database Tools, Managing Remote Computers, And More

If you want to apply static analysis to your databases, connect to remote computers, find out more about the Entity Framework, or just check into some cool podcasts for your daily commute, then you'll want to read more about these latest tools and resources.

Scott Mitchell

MSDN Magazine July 2009

...

This month the Toolbox column takes a look at database logging, Joel Spolsky's blog, printing code projects, and ASP.NET reading.

Scott Mitchell

MSDN Magazine May 2008

...

Powerful Text Editing, Web Testing in .NET, Extended Unit Testing, and More

James Avery

MSDN Magazine February 2008

...

Read more!

Data Points: Common Table Expressions

In T-SQL, views and derived tables serve similar purposes and have their advantages. But when working with SQL Server 2005,there's a third option--using Common Table Expressions (CTEs). John Papa explains.

John Papa

MSDN Magazine October 2007

...

Here the author introduces SQL Server Data Services, which exposes its functionality over standard Web service interfaces.

David Robinson

MSDN Magazine July 2008

...

Read more!

Popular Articles

Advanced WPF: Understanding Routed Events And Commands In WPF

See how routed events and routed commands in Windows Presentation Foundation form the basis for communication between the parts of your UI.

Brian Noyes

MSDN Magazine September 2008

...

Chris Tavares explains how the ASP.NET MVC Framework's Model View Controller pattern helps you build flexible, easily tested Web applications.

Chris Tavares

MSDN Magazine March 2008

...

We introduce you to the benefits of building composite applications with the Composite Application Guidance for WPF from Microsoft patterns & practices.

Glenn Block

MSDN Magazine September 2008

...

This article introduces 10 development tools that can increase your productivity, give you a better understanding of .NET, and maybe even change the way that you develop applications. The tools covered include NUnit to write unit tests, Reflector to examine assemblies, FxCop to police your code, Regulator to build regular expressions, NDoc to create code documentation and five more.

James Avery

MSDN Magazine July 2004

...

C# allows developers to embed XML comments into their source files-a useful facility, especially when more than one programmer is working on the same code. The C# parser can expand these XML tags to provide additional information and export them to an external document for further processing. This article shows how to use XML comments and explains the relevant tags. The author demonstrates how to set up your project to export your XML comments into convenient documentation for the benefit of other developers. He also shows how to use comments ...

Read more!

{End Bracket}

The World-Wide Telescope

Jim Gray and Alexander Szalay

Someday all scientific literature and data will be online and accessible to everyone everywhere. The astronomy community has made unusually good progress toward this vision of online science in addition to addressing the associated challenges of data publication.

Much of astronomy involves comparing data from many instruments (from different parts of the electromagnetic spectrum) taken at different times. The Crab Nebula provides a good example of the temporal and multispectral nature of astronomy. The Crab Star supernova was first observed on July 4, 1054. Now we see it as the Crab Nebula—a gas cloud expanding at relativistic speeds. The cloud's core appears to be a black hole. Matter falling into this hole emits two energy beams that illuminate the cloud. Looking at this system in the X-ray spectral band shows the beams. Looking at the system in the optical and infrared spectral bands shows the gas escaping from the black hole. Each provides complementary information that together presents a fairly complete picture.

Astronomy obeys Moore's law: it is producing about two times more data each year. Current instruments typically produce nearly a terabyte per night. Managing huge data archives and processing complex data are now among the major astronomy challenges.

The Sloan Digital Sky Survey (SDSS) is a good example. It is a 5-band optical survey of the northern sky, observing about 400 million sources as images and 1 million with spectra. These spectra allow detailed studies of large star and galaxy populations. The traditional way of accessing this data is to place it in files and let users FTP the files they want to their local systems for analysis.

We built an online catalog of the SDSS data as a Web-accessible database, along with visual tools to analyze the data (SkyServer.sdss.org). The result is a SQL Server™ database with approximately 14 billion rows. It gives full GUI and SQL access to the SDSS data. Now everyone can use one of the world's best telescopes. The site has been a big success—about 10 percent of the visitors are students using online courses, but the main users are astronomers analyzing the available data.

SkyServer must support many browsers running on many platforms, so we took a thin-client approach in which most processing is done on a server that produces standard HTML. Much of the server-side logic is implemented in T-SQL stored procedures. The Web services rendering images are coded in C#. The database schema is self-documenting, and the design allows users to plug their catalog and image data into a spatial search framework and Web service. A C# variant of the spatial data search is part of the SQL Server 2005 samples. You can download a personal copy of all the SkyServer code (Web site and database) along with a version of the database from SkyServer.org.

SkyServer has quite a few Web services to give users programmatic access to the data and analysis tools, a classic service-oriented architecture. But some of the astronomy queries run for hours, so we set up a system to let users submit long-running jobs (the Catalog Archive Serve Jobs System, or CASJOBS). We also allowed users to create personal databases (MyDB) near the server. MyDB stores intermediate results and uploaded user data, allowing users to do multistep analyses on huge datasets. The article "Batch is back: CasJobs, serving multi-TB data on the Web,", describes this system.

The International Virtual Observatory Alliance is a grassroots group of astronomers who want to federate world-wide astronomy data, cross-index it with the literature, and provide analysis tools to the community. The SDSS data is now part of this federation. A user can point to a SkyServer object and find all the literature on that object, as well as all other public data sets about that object. This is all architected using Web services defined by the IVOA. The IVOA is also defining a standard schema and ontology for astronomy data (www.ivoa.net/Documents).

We are now much closer to reaching the goal of the World-Wide Telescope federating all the world's astronomy data and literature. OpenSkyQuery.Net offers a prototype federating 29 archives. The astronomy community is well on the way to building the next generation of this design.

Jim Gray works at Microsoft Research and focuses on eScience scaleable systems and databases. He has been active in building online databases like terraService.Net and SkyServer.sdss.org. See research.microsoft.com/~gray.

Alexander Szalay is a Professor at the Johns Hopkins University, both in the Department of Physics and Astronomy, and the Department of Computer Science. He is the Project Director of the NSF-funded National Virtual Observatory. See www.sdss.jhu.edu/~szalay.