Automating the Build Process
An automated build process compiles, deploys and then runs build verification tests (BVTs) against the latest source code for a project at regular, predetermined intervals. Then a “build report,” which details the success or failure of the build process, is disseminated to the project stakeholders. The build report is analyzed to determine what areas of the project need attention and/or if the project should be rolled back to an earlier version/build.
The power of an automated build process is it can be scheduled to be run during “off hours” so it can help ensure the stability of the project without taking cycles directly away from the development time. This topic provides an overview of the build process, describes how build verification testing fits into the build process, describes aspects of functional testing used during build verification testing and provides information about the importance of automating the build process.
Before looking into the specifics of testing, it’s important to consider the context of how testing fits into the overall build process. All of Microsoft’s product groups operate from the philosophy that the build process is the heartbeat of any software project. This approach is also used by many of the world’s top consulting firms that build mission critical solutions on top of the Microsoft software stack. Without a steady heartbeat and a regular check of it, it is impossible to know the health of the project. To ensure the product groups have a continuous insight into the overall health of the project the build process is run daily, typically at midnight. The steps below summarize a typical automated build process:
- Obtain latest build of source code from source code repository.
- Compile the source code.
- Pack up binaries for deployment – typically using scripts or Microsoft® Windows® Installer files.
- Deploy project to a test server.
- Automatically run a full set of build verification tests (BVTs).
- Publish a detailed build report to members of the project team.
|BVTs are functional tests that exercise the main features of the solution to test its quality. You need to ensure your BVTs are comprehensive enough to gauge the build quality, yet small enough to execute within the allocated daily build time!|
|You should also treat any deployment, un-deployment, configuration and installation scripts or processes as part of the software project for testing purposes. Both the operations of the project, as well as the deployment and configuration of it, should be tested.|
This process is typically completed by early morning around 6 A.M., which enables the first members of the team to start work on the new day’s build. If one or more of the BVTs from the previous night failed, then it is the team’s responsibility to fix it as soon as possible.
Following a daily build process has many advantages for the project. First, it provides a regular heartbeat (made up of the daily build plus the automated BVTs). Second, using BVTs forces integration with systems, this is a tricky task and doing this early in and of itself reduces project risks. Due to the time required to complete them, stress and performance testing are typically performed outside of the daily build process. Stress and performance tests are typically scheduled to be performed on milestone build in the project.
The daily build process can be and has been used very effectively on BizTalk solutions. However, you need to ensure tasks that are typically left to the end in projects are done iteratively from the start. For example, deployment in BizTalk Server is certainly non-trivial. It is important automated deployment scripts be developed up front. If you do not do this, you’ll end up manually deploying and un-deploying the solution many times throughout the project, which will cost you more time in the end. There are tools available to drive the daily build process; Visual Studio Team System and Team Foundation Server are the primary choice for many people. MSBuild script(s) may be used to drive the steps in the build process. Another alternative is the open source CruiseControl.NET tool, which is available at http://go.microsoft.com/fwlink/?LinkId=116093.
|Use of this tool is not supported by Microsoft, and Microsoft makes no guarantees about the suitability of this programs. Use of this program is entirely at your own risk.|
For more information about automating testing using Visual Studio Team System, see the topic "Testing Tools Tasks" in the Visual Studio Team System online documentation at http://go.microsoft.com/fwlink/?LinkId=120226. For more information about automating the build process using Visual Studio Team System, see "Managing Builds with Team Foundation Build" in the Visual Studio Team System documentation at http://go.microsoft.com/fwlink/?LinkId=116130.
Build verification testing usually comprises the following elements:
- Unit Tests - As unit tests are typically the first tests to be developed, ideally they should be created before the code has actually been written if you are truly using a test-driven development approach. By adding them into BVTs during the early stages of a project, you provide at least some code coverage immediately. As the number of functional tests and hence test coverage grows, you may opt to remove unit tests from the BVTs.
- Smoke Tests - End-to-end functional tests that test the basic functionality of your solution. If these fail, something is seriously wrong. These can usually be run relatively quickly.
- Functional Tests - These also target end-to-end scenarios, but in this case they test all the scenarios in the project. For large projects it may make sense to categorize your functional tests further (e.g. to enable a particular component to be tested quickly, reliably and in isolation). These functional tests should be locked down after you have signed off on your solution as being functionally correct.
- Regression Verification Tests - Every time a solution bug is found and fixed, regression verification test cases should be added to verify the bug is fixed and it is not reintroduced later in the project lifecycle. These tests will typically be edge cases that were not covered in the functional test cases. You should expect your regression verification tests will increase in number even after the solution has gone live, if new bugs are discovered and fixed.
On very large projects, you may need to make your BVTs a subset of the full functional test suite (due to length of time they take to execute). For smaller projects, BVTs will constitute the entire set. Obviously, if you are going to integrate the BVTs as part of your daily build, then the whole process needs to be automated. In the rest of this section, we will focus on how you can use BizUnit and other tools to accomplish this.
In the context of BizTalk applications functional tests, test a specific end-to-end scenario. When performing this type of testing, it is useful to imagine BizTalk Server as a black box you test functionally from an external perspective. For example, a test may involve feeding an input message (with known values) to a receive location end point (e.g. URL, FTP location, whatever your choice of transport is). The test would then monitor the correct number of messages with the correct output that are produced on the send side. This may sound relatively straightforward, but when you consider some scenarios require inputs to come in a certain order and at a certain time and you compound this with other solution requirements, such as, when recording tracking data in BAM, it becomes clear this is a classic case where a tool and framework can be used to orchestrate this.
It is critical functional testing is designed to cover all the possible paths through your solution. This should include not only those scenarios you expect in production, but also the failure paths and exception handling paths you have implemented but hope never to use – one phrase commonly used to describe this is testing for the “bad day scenario.” You should ensure all orchestrations, all permissible message types, and all code branches are exercised by your functional test suite. The following sections describe developing positive and negative functional test cases to cover all code paths.
For more information about functional testing and the other testing categories should be implemented before placing a BizTalk Server solution into production, see the topic “Checklist: Testing Operational Readiness” in the BizTalk Server Operations Guide at http://go.microsoft.com/fwlink/?LinkId=116064.
- It is important when performing positive tests to ensure all combinations of messages, pipelines, orchestrations and endpoints are passed through the solution to make sure all the message flows are exercised. To ensure you test all code paths will likely require you process different messages with different content.
- When testing, use the transport type will be used in production. Unfortunately all too often functional testing is performed only using the file adapter when some other transport will be used in production. Adopting this approach is setting you and the overall project up for failure later on.
- Validate the payload of all messages that are sent out from the system. If the messages are XML, you should validate their schema and key fields in the message using XPath expressions.
- Validate any tracking data stored in BAM (if used) or any other data should be left in external data repositories is accounted for.
- Test the execution of all Business Rule Engine (BRE) policies and rules if your solution uses BRE.
- Ensure you test the handling of invalid messages through your system. You should verify your chosen strategy (to reject them before they come into BizTalk Server or to suspend them) has worked correctly.
- When testing the handling of invalid messages, ensure you test any receive-side error handling orchestrations have been implemented to handle suspended messages.
- Ensure your failure scenarios cover all exception blocks in your orchestrations. Failing to test this adequately is a common problem.
- If you are using long-running transactions with compensation behavior, test these scenarios very carefully. This is another area where inadequate testing will incur serious consequences in a production environment.
- Ensure failures are logged correctly in the appropriate error logs.
In order to do all of this efficiently and effectively, invest the time upfront to automate testing. Manual testing is time consuming, error prone, and expensive (in terms of time and cost). Every time you perform a manual test pass, you add another batch of tasks that have to be handled by project resources (i.e. people in the team). By automating this up front, you are able to get a return on the upfront investment that is required to develop the tests every time they are run. Good functional tests should execute quickly and efficiently and be repeatable; each test should also be autonomous (i.e. independent of any other, adopting this approach enables you to run multiple tests sequentially as a test suite). The functional tests should always produce the same result, as poorly written functional tests or poorly written code will result in different tests results between test runs, leading to confusion and wasted time investigating the root cause of the failure.
It is important to minimize the development effort required to write each functional test. Usually the more expensive it is to produce something (in terms of development time), the fewer test cases you are likely to end up with. This means you will have a lower level of test coverage over your code. By utilizing a test framework, you can develop test cases quicker and easier and, hence, make it easier to get full code coverage. Most good test frameworks use a declarative approach to defining tests (i.e. the configuration for a test is stored in a configuration file which is typically an XML file). Utilizing a good test framework enables you to develop a full functional test suite in an agile and reliable manner and avoids having to “reinvent the wheel” over and over, so to speak.