As the complexity of modern software development lifecycles increases, it’s important to have a comprehensive monitoring solution for your continuous integration (CI) pipelines so that you can quickly pinpoint and triage issues, especially when you have a large number of pipelines running.

Datadog now offers deep, end-to-end visibility into your TeamCity builds with the new TeamCity integration for CI Pipeline Visibility, helping you identify bottlenecks in your CI system, track and address performance regressions, and proactively improve the efficiency of your CI system.

Making data-driven decisions to increase the performance and reliability of your pipelines will help you improve end-user experience by allowing your team to push code releases faster and with fewer errors.

In this post, we’ll show you how to:

• Integrate TeamCity with CI Visibility
• Investigate pipeline failures to fix erroneous builds

Integrate TeamCity with CI Visibility

To configure the TeamCity integration with Datadog CI Visibility, first download the Datadog CI plugin on the TeamCity server. Then, ensure that the last build of your build chains is a composite build. Build chains in TeamCity map to pipelines in Datadog, and individual builds map to pipeline executions.

Add the following parameters to your project:

• datadog.ci.api.key: your Datadog API key
• datadog.ci.site: datadoghq.com
• datadog.ci.enabled: true
  

Once you’ve enabled the integration, data from your TeamCity pipelines will automatically flow into Datadog. If you navigate to the Pipelines page, you can see TeamCity pipelines alongside any other providers you may have instrumented with CI Visibility.

Investigate pipeline failures to fix erroneous builds

After you enable the TeamCity integration in CI Visibility, you can use the Pipeline overview page to get a high-level view of the health and performance of your TeamCity build chains, with key metrics such as executions, failure rate, build duration, and more.

Say you’re an engineer at an e-commerce company where one of the checkout services for your primary application is undergoing a major revamp under a tight deadline. After pushing new code, you notice that your builds are extremely slow—much slower than normal. You can go to the Pipelines page in CI Visibility to confirm if your particular pipeline is experiencing high build durations. Then, you can click on the build chain from the Pipeline overview page to investigate the pipeline in more detail.

At the top of this Pipeline Detail view, you can see the status of the last build, with a link to the build chain in TeamCity. Below that are timeseries widgets illustrating the total number of builds, the error rate, build duration, and other key metrics that can help you determine when the build chain began to experience errors. In this case, you see the error rate spiking repeatedly over the past several days.

The Job Summary gives you more granular information about your build chain, such as which specific jobs in this pipeline failed the most, which ones took the longest, and which jobs have experienced performance regressions compared to the previous week. Information like this can help you identify the areas in your CI system where optimization will result in the greatest performance gains.

To investigate further, you can scroll down to see the individual builds for this pipeline. If you click on an execution, you can see a flame graph view that visually breaks down the pipeline execution into the individual jobs that ran sequentially and in parallel.

The flame graph shows you each build’s respective duration broken down by job and, if the build was erroneous, the exact parts of the build that failed. This can help you pinpoint problematic jobs that may be at the root of a failed build.

The Info tab shows you repository and commit information along with other git metadata, so you can easily see the source of each build. To investigate further, you reach out to the team member who pushed the commit for this build and discover that the issue is caused by a typo. (We strongly recommend that customers use a TeamCity username style that contains author email, so that Datadog can automatically detect git author email addresses and correlate commit information to pipeline data.)

Once resolved, the build chain functions without error so you can build and test successfully, and release your updated checkout service to customers on time.

Understand and optimize TeamCity build chain performance

CI Visibility support for TeamCity is now generally available, giving you deep visibility into your build chains so you can troubleshoot failed builds, identify performance regressions faster, and increase your release velocity.

For more information, see the Datadog documentation and blog post on the TeamCity Agent integration.

If you’re new to Datadog, sign up for 14-day free trial.

To support our growing team, offer faster release cycles, develop additional built-in integrations, and provide better and faster support to all of our customers, we have decided to increase prices for both new licenses and renewals of existing licenses of TeamCity On-Premises.

The increase in pricing will affect only TeamCity On-Premises and not TeamCity Cloud, which follows a different licensing model. New prices for TeamCity On-Premises will come into effect on September 1, 2023. You can find both new and current prices on the TeamCity Pricing page.

Renew existing licenses at the current price

Typically, we only permit the renewal of TeamCity On-Premises licenses up to one year in advance. This means that if the maintenance periods of your existing TeamCity Enterprise or TeamCity Build Agent licenses are set to expire within one year from today, you can renew those licenses for an extra year.

However, to help provide our customers with more time to benefit from the current renewal prices, we are introducing a temporary offer. If your TeamCity Enterprise or TeamCity Build Agent licenses have a maintenance window that expires within two years from today, you will be able to renew those licenses for an additional year at the current price. This offer applies to only those orders placed before August 31, 2023.

For example: If the maintenance period of your TeamCity Enterprise or TeamCity Build Agent licenses is valid through April 30, 2024, you have the option to prepay now and extend your licenses at the current rates for an additional two years until April 30, 2026.

While this offer requires an upfront payment, it allows you to renew at the current price for one additional year.

We understand that a price increase is not something any of us like. However, we’ll continue to do our best to minimize these.

We will be reaching out to all of our customers over the next few weeks regarding this update.

Thank you for your support.

The integration with Let’s Encrypt and the two-factor authentication for custom groups are here to further strengthen TeamCity’s security features.

With this release, we’re also rolling out the GitHub App connection, the ability to send custom Slack messages right from a running build, and more.

Let’s take a closer look at what TeamCity 2023.05 has in store for you.

Dark theme

We realize how important it is for developers to feel comfortable with the tools that they use on a daily basis. That’s why we’re happy to release one of the most upvoted features in TeamCity – the dark theme.

The dark theme can provide a more comfortable and productive working environment. It reduces eye strain, helps improve your focus by eliminating visual distractions, and extends your device’s battery life. Last but not least, some users find it more visually appealing.

The dark theme in TeamCity is available both for the Classic and Sakura UIs. In addition to the “Light” and “Dark” options, you can also choose “System theme” to automatically apply a theme that fits your current OS settings.

To switch between the themes, head over to your profile settings and choose your preferred theme:

You can also easily switch between the themes via the “Theme” drop-down that’s available right in the header:

Have you started using the dark theme yet? Feel free to share your feedback in the YouTrack issue.

Agent terminal

TeamCity 2023.05 comes with a convenient way to open remote terminals to agent machines directly from the TeamCity UI. With the help of remote terminals, you can view agent logs, check installed software, and debug specific agent issues.

This functionality comes especially handy when you need to troubleshoot any issues on an agent. Thanks to remote terminals, you don’t have to worry about maintaining SSH keys for connecting to agents. Mac, Linux, and Windows (with PowerShell) are supported.
To open the terminal, head over to Agents in the TeamCity header, select an agent, and click Open terminal at the top of the agent page:

Multi-node setup enhancements

With this release, we’re implementing an improved requests distribution logic that efficiently distributes the load between main and secondary nodes.

By making nodes fully interchangeable, we’re improving TeamCity’s scalability. All TeamCity nodes can now participate in balancing out user requests, which means the workload produced by web UI users can be evenly distributed between all the nodes.

The new distribution logic provides a smoother and more reliable experience for all and helps to ensure that the impact on users is minimal in the event of planned maintenance or an unexpected failover.

Assign the VCS polling responsibility to multiple nodes

TeamCity users no longer have to rely on a single node to handle the “VCS repositories polling” responsibility. With v2023.05, this function can now be assigned to multiple nodes, resulting in a more balanced workload, faster detection of new commits, and faster starting of builds. As a result, you can build, test, and deploy with greater efficiency, reducing delays and enhancing overall productivity.

Security features

At TeamCity, security is our top priority. To further strengthen the product’s resistance to data breaches and potential business disruptions, we’re implementing the following features.

Integration with Let’s Encrypt

Starting from version 2023.05, TeamCity integrates with Let’s Encrypt, a free and automated certificate authority that provides digital certificates for enabling HTTPS on websites. TeamCity can contact this certificate authority to automatically issue and set up a valid certificate. The certificate will then be updated automatically, without any actions required from the user.

Read more about TeamCity’s integration with Let’s Encrypt in our documentation.

Two-factor authentication

Previously, the mandatory two-factor authentication mode used to affect the whole server. With this release, we’re making it possible to create custom user groups where the 2FA mode will be mandatory even if the global mode is set to “Optional”.

GitHub App connection

It’s now possible to authorize TeamCity to work with GitHub as a GitHub App. With this new feature, you can run various actions and use the GitHub API without having to create separate service accounts or act on behalf of a user.

To configure GitHub App as a connection, head over to Project Settings | Connections | + Add connection and select “GitHub App” from the dropdown menu.

After that, if you don’t already have a GitHub App, follow the TeamCity instructions to create and install a new App with the required permissions.

One of the biggest benefits of using the GitHub App connection is that instead of accessing GitHub under a user’s credentials, TeamCity will use non-personal, short-lived, refreshable tokens issued for a GitHub App installation in the user’s GitHub organization/account. This comes in handy when a user leaves the organization or their permissions change.

Additionally, you can subscribe to various hooks from GitHub without the necessity to install any additional plugins. The connection works right out of the box, making the integration between TeamCity and GitHub seamless.

Head over to the docs to read more about how to configure a GitHub App connection in TeamCity.

Improved support for draft pull requests in the Pull Requests plugin

Earlier, in TeamCity Cloud version 2022.12, we introduced the ability to configure the Pull Requests build feature to ignore GitHub draft pull requests.

Now, this functionality is available to all TeamCity users. In this release, you can opt to ignore GitHub draft pull requests using the Pull Requests build feature. To do so, simply select the “Ignore Drafts” option in the feature settings. Draft pull requests will be ignored by TeamCity until their status changes.

Normally, the Pull Requests build feature retrieves information for GitHub draft pull requests, and runs builds on them by default. When a draft pull request is being built, the build page shows the pull request number with the “Draft” status and icon next to it.

Customizable notifications

TeamCity already offers the ability to send email and Slack notifications when triggered by certain events. With the 2023.05 release, we’re extending this functionality with the ability to send Slack messages and email notifications via service messages.

With the latest update, TeamCity has added the ##teamcity[notification...] service message that enables users to send email notifications, Slack direct messages, and Slack channel updates. It’s now also possible to send notifications to multiple people at once.

These messages are secured by built-in safety measures to prevent them from being delivered to the wrong recipient and to block links to untrusted external resources.

One scenario in which this functionality can be helpful is when you want to send a notification when certain conditions are met, or something outstanding happens, for example, when an ongoing deployment reports an error that should be reviewed while the build is still running.

Add and remove build tags via service messages

In addition to sending custom notifications, you can also add and remove build tags with the help of service messages.

Tagging builds provides you with the ability to label and systemize builds by certain criteria. For instance, you can use build tags for tagging the reasons why builds fail and then quickly finding builds with the same failure reason.

Head over to the documentation to learn more about how to set up build tags. Or, if you have already started using the feature, feel free to share your feedback in this YouTrack issue.

Podman support

With the 2023.05 release, we’re introducing support for Podman, a container engine that allows users to manage containers without requiring root privileges. It is fully compatible with Docker commands, making it easier to switch between engines.

Over to you

For the full list of new features, please make sure to check out our What’s New section in the documentation.

As always, we’re looking forward to your feedback! Feel free to submit a request via the support form or simply leave a comment under this blog post.

Happy building!

CI/CD servers are high-value targets for attackers because of their central role in critical development processes. They provide access to source code, a valuable asset for software companies, and can deploy code to production environments, creating serious risks if not adequately secured. Even a single vulnerability can enable attackers to compromise the supply chain, inject malware, and seize control of systems. 

According to “The State of Software Supply Chain Security 2023”, this has led to a rise in supply chain attacks since 2020, and 57% of organizations have suffered security incidents related to DevOps toolchain exposures. 

To avoid data breaches and business disruptions, securing CI/CD servers should be a top priority. Furthermore, Google’s “2022 Accelerate State of DevOps Report” suggests that implementing proper security controls can have a positive impact on software delivery performance.

In this whitepaper, we present 9 effective ways to prevent a supply chain attack on your CI/CD server, providing practical guidance and best practices to help you strengthen security and protect critical development processes.

By implementing these strategies, you can minimize the risk of a supply chain attack and ensure the integrity, availability, and confidentiality of your software supply chain.

Download the whitepaper

In this case study, we delve into how Gearbox Software leverages TeamCity to optimize its development process and achieve remarkable results.

Streamlining development with TeamCity

Gearbox Software found a powerful ally in TeamCity. By integrating TeamCity into their development pipeline, they were able to automate their build, test, and deployment processes effectively.

TeamCity’s flexible and intuitive interface allowed Gearbox Software’s developers to set up build configurations effortlessly and manage their projects efficiently.

One of the competitors had a very unfriendly UI. The feeling that people should have when they come into a CI/CD system is that it is very sturdy, it won’t break when I try to do anything in it. I think TeamCity has a very well-polished UI. You have confidence that it is a well-working system when you navigate it.
– Steve Fortier, Lead Release Engineer

Knowledge sharing made easy

Gearbox Software faced the challenge of setting up new projects from scratch, leading to a divergence between projects and the need for subject matter experts for each project. This hindered knowledge sharing, caused bottlenecks, and increased the risk of errors.

However, TeamCity helped solve this issue by providing a streamlined setup process, reducing divergence, promoting knowledge sharing, and improving overall development efficiency.

Increased productivity

After implementing TeamCity as their CI/CD solution, Gearbox Software experienced significant improvements in its development process. With the ability to create templates, the team was able to share resources across projects, leading to greater familiarity and increased productivity.

This streamlined approach allowed the team to focus on effective development practices, resulting in enhanced overall efficiency.

Significant savings in terms of EC2 instances

By transitioning to TeamCity, Gearbox Software resolved a bottleneck in their previous CI solution where each build had to compile the editor, consuming valuable time of up to an hour.

With TeamCity, the team separated the editor build into a standalone process that is reused across all builds. This optimization resulted in notable savings in EC2 instances as redundant compilation of the same code was eliminated, enhancing efficiency and reducing resource consumption.

Configuration as code at Gearbox Software

When Gearbox Software started using Kotlin for project configuration, the team was very excited. Even those with little experience with Kotlin were able to figure it out and start using the programming language quickly enough. “There’s a learning curve, but generally, the vibe was positive,” said Steve Fortier.

Read the full case study

These are now available in our new learning center right within TeamCity Cloud. That means you can find answers to your most burning TeamCity-related questions and access video guides without ever having to leave the platform.

Here are some of the questions that the learning center can answer for you:

• What’s the difference between checkout rules and trigger rules?
• How to run command line scripts?
• How to work with artifacts? 
• And many others.

How to access the learning center

To access the learning center, log in to your TeamCity Cloud account and navigate to the Learn tab in the top right menu. 

The video tutorials are grouped by topics, including integrations, source code repositories, build agents, tests and build chains, and the Kotlin DSL. Thanks to the tags, you can conveniently see which topics a particular video covers.

How can the learning center help you?

TeamCity is a powerful platform that can be fully customized according to your organization’s needs. However, learning to work with a new CI/CD platform might seem overwhelming at times. 

TeamCity’s learning center provides you with hands-on explanations of TeamCity’s key concepts and gives you an overview of what the platform is capable of. With the help of the video tutorials, you can learn how to create your first build pipeline, integrate TeamCity and Github, work with pull requests, and get started with the Kotlin DSL.

What’s next?

Video guides are just the beginning of TeamCity’s learning center. We’re planning to add more videos on various topics to help our users take their CI/CD skills even further.

We’re also enriching the learning center with interactive guides. The guides will walk you through TeamCity, introduce you to key TeamCity concepts, and help you master the platform.

More learning resources

To level up your CI/CD skills and boost your organization’s performance, make sure to check out our blog where you can find helpful posts and how-to articles on various topics.

TeamCity’s documentation will provide you with further guidance on how to set up a robust CI/CD process in your organization.

As always, feel free to reach out to us if you have any questions. We’re just an email away.

Happy building!

Do you use feature branches or Git flow to ensure code quality?

The powerful combination of JetBrains Space Git flow with TeamCity offers a complete solution to help you achieve better code quality and keep your main branch green with stable builds. 

Space provides you with Git hosting, code reviews, and quality gates, while TeamCity offers a build pipeline. The native integration between tools will save you time and effort by ensuring you have a complete and well-integrated flow with a unified UI. 

Adopting the Space Git flow with TeamCity is easy and allows you to keep your main branch stable while controlling changes automatically. 

In this blog post, we’ll introduce the combo of the Space Git flow and TeamCity, and guide you through the process of using it in your project.

Get started with Space for free

What Is the Space Git Flow, and Why Use it?

The Space Git flow is a branching strategy that is similar to GitHub flow, but with a greater emphasis on safety when introducing changes to the main branch and the ability to scale to large projects and teams.

Getting started with the Space Git flow is easy and doesn’t require going all-in, as you can mirror your existing Git repository without having to migrate, and you can switch back any time.

Using the Space Git flow integrated with TeamCity allows you to:

• Achieve higher-quality code and a stable main branch by configuring quality gates based on TeamCity build status for merge requests.
• Introduce a code review process your team will love, integrated with IntelliJ IDEA and available on the go with Space iOS and Android apps.

Read this article to learn more about the Space Git flow.

How to Use the Space Git Flow With TeamCity

Watch this 3-minute video to see how the combination of the Space Git flow and TeamCity works in a project:

Read on to learn about the flow in more detail.

Mirror or host your Git repository in Space

Start by creating bi-directional mirrors of your repository from GitHub or another Git service and link it to Space in a few clicks. 

You can also use Space to host your source code privately. If you host it in Space, you can take advantage of the web-based interface and JetBrains IDE integration for a seamless development experience.

Protect your main branch from accidental pushes  

To prevent accidental pushes to the main branch, you can set up branch protection in Space. This helps you protect repositories from unauthenticated commits, restrict who can push or merge changes into the branch, and prevent accidental branch deletion.

Once you’ve set up branch protection, the only way a change can make it to main is through a merge request.

Create a feature branch

To start working on a new feature, you’ll need to either clone the repository to your machine, or use remote development with the IDE hosted on a virtual machine in the Space cloud.

With the Space Git flow, you make changes in a separate feature branch and then request a code review before merging it back to main. From your IDE, you can make changes, add a commit message, and then push this branch back to Space. 

Create a merge request

Create a merge request to begin the process of moving your new code into main. You can do this from within your IDE using the native integration, or you can find your branch in Space and create a merge request from there.

You can create a title and description for your merge request and see the commits that will be part of it. The quality gates configured in Space require at least one person to review these changes.

Quality gates are a mandatory and customizable set of conditions that need to be fulfilled before merging. Apart from code reviewer’s approval, you can set up a Space Automation job and a TeamCity check as gates.

Space will require a review based on code ownership

In your repository, you can create a special file named CODEOWNERS that specifies who is responsible for specific folders and files. Later, you and your colleagues will have the option to select a code owner to review your changes as a part of quality gates.

Wait for your team to review the changes 

Now you wait for your colleagues to review the changes. They can add comments and suggestions, or they can simply approve the request. With turn-based code reviews, Space makes it easy for both the author and the reviewer to understand whose turn it is to take action.

Space allows you to collect your comments as drafts and send them in batches so as not to overload your colleagues with unnecessary notifications.  

After you’ve agreed on changes with your reviewer, you can proceed with merging them.

Ensure your build is green 

With Space and TeamCity, there are a few options that you can use to make sure your build is green before merging your changes. You can use either or both of them for extra security.

Run CI/CD checks

After you create a merge request, Space Automation and/or TeamCity CI jobs are triggered to validate the changes by building the code and running tests. As part of quality gates, the changes will be merged into main only if a CI/CD server can successfully build the feature branch. 

The checks can be set and completed by both Space Automation and TeamCity.

These checks, however, only ensure the new code is valid and integrates well with the main branch at the time of branching. With Safe Merge, validation is done by attempting to integrate the new code with the very latest code from the main branch.

Run Safe Merge 

You can use Safe Merge before merging the changes directly into main. Safe Merge acts like a time machine – it runs a preview of what your main branch would look like with the changes. 

This can be especially useful for large projects because while you were working on a feature branch, the main branch could have received changes that conflict with your work. Safe Merge allows you to catch these conflicts before actually merging the branches.

Space creates a temporary commit with the latest changes, which can then be used to perform the required quality checks using an Automation job or TeamCity build.

In TeamCity, you’ll see Space trigger a special build, and TeamCity will report the build status back to Space when it’s finished. The changes are merged into the main branch only if the build succeeds. Otherwise, the merge is rejected.

Using the Space Git flow together with TeamCity ensures that changes to your code base are always reviewed. And thanks to Safe Merge, you’ll always have a main branch that compiles and can be deployed successfully.

How to Get Started With the Space Git Flow and TeamCity

If you already use TeamCity, getting started with the Space Git flow is easy. 

• Create a Space organization to get started.
• Check out this step-by-step tutorial to configure Space Git flow and integrate it with TeamCity. 
• Request a demo to ask your questions or get help tuning the flow to your specific process.

Get started with Space for free

What do you think about the combination of the Space Git flow and TeamCity? We’d love your feedback in the comments below.

First, some numbers

On a typical working day, the load on the build server increases after 14:00 CET and starts to drop after 20:00. This is the rush hour, when the number of queued builds is constantly high. For example, take a look at the statistics from February 22 to March 3:

The steps below played a significant role in making the server stable under this load.

Database hardware upgrade

The cheapest and the most straightforward step was probably upgrading the MySQL database used by the build server. When we initially moved the build server to AWS, we used a MySQL server instance with 16 cores and 64 GB of memory. However, it wasn’t enough to sustain the load. The average CPU load was constantly around 75%, so we switched to a larger machine with 32 cores and 128 GB of memory.

The challenge was to perform this upgrade without server downtime and ideally not over the weekend. We found a way to switch all TeamCity nodes from one database to another at runtime without restarting any nodes. As a result, the migration on the morning of December 5th was smooth and non-disruptive. The average CPU usage dropped to 45%, which will do its part to enhance build server performance.

The 4th node

For almost all of 2022, the build server cluster consisted of 3 nodes. By the end of the year, we decided to add a 4th node. The main reason for this decision was to have extra resources to handle the load when we need to bring down nodes, like for maintenance. 

This happens from time to time, and is usually because of bugs. For a critical bug, we restart nodes one by one and apply hotfixes. This means until all nodes are restarted (which can take quite a while), our cluster is down to only 2 nodes which may struggle to sustain the workload our build server is dealing with.

Before the end of 2022, we added the 4th node to the cluster. All 4 TeamCity nodes are identical: 64 cores and 500 GB of RAM for each.

Round robin

A round robin of user sessions across all TeamCity nodes was in development for quite some time. It was launched in 2021 and has progressed at different paces over the last couple years.

While all TeamCity nodes appeared and behaved as regular TeamCity servers, some functionality was still missing. For instance, if a user landed on a secondary node, they would not be able to see agent logs or build process thread dumps.

It took some time to find all of the limitations and eliminate most of them. By mid-2022, we felt confident that secondary nodes would act almost identically to the primary node. At this point, we launched an experiment where a group of volunteers from the TeamCity team started using the secondary node to carry out their daily work.

Several months later, in December 2022, we expanded this round robin for all build server users. This caused some confusion, as some UI features were still unavailable on the secondary nodes, but overall it went well. If you paid close attention, you’d notice this small selector in the footer:

This selector allows you to check your current node and switch to another one. If certain functionality does not work as expected on a secondary node, switch to the primary node-1.

Fixing the event lag

When something important happens with major TeamCity objects (such as builds or projects), the node where the object’s state was changed publishes something called a “multi-node event”. This event is transmitted to all of the other nodes in a cluster. Depending on the event type, these other nodes may update their own in-memory caches and objects.

As time went by, our build server ran more and more concurrent builds, processed bigger build queues, and handled more agents. This resulted in a significant increase in events and revealed bottlenecks and inefficiencies in the code that processes these events, which usually looked like the following:

We called it the “event lag”. As you can see in the image above, linux-2 and linux-3 nodes cannot process all of the incoming events in time, and the backlog of unprocessed events grows rapidly. At a certain point, nodes are unable to recover and have to be restarted. The event lag was a critical issue that kept us from routing users to secondary nodes – the secondary nodes would simply miss the most recent data there.

This major issue had no quick and easy fix. Instead, we had to rewrite our code to process most of the events in batches and modify the processing logic for other cases. In the end, we managed to resolve this problem. Now, even a hot day on a TeamCity build server looks like this:

Improving build queue processing performance

The TeamCity build queue bottleneck was previously described in this blog post.  

We’ve managed to solve most of the issues mentioned there. Since then, the queue processing algorithm has become more parallel, and we’ve improved its performance even further. TeamCity can now process 20k builds in the queue without significant delays. The time required to process a queue this large is still 2-3 minutes, but the previous time was much higher than that. 

Despite the progress that has been made, we’re not calling it a day yet. We will keep looking for a way to further speed up the queue processing.

Change collecting on multiple nodes

Previously, we had a dedicated node (node-3) to perform all “change collecting” operations whether they were caused by a commit hook, regular VCS polling, or the start of a build. 

As a result, this node was sometimes overloaded with all of the requests to collect changes in different VCS roots, especially when many large build chains were triggered simultaneously.

In mid-February 2023, we enabled an experimental mode in which “change collecting” operations can be performed by any node in the cluster in parallel, as long as the single VCS root is being processed by one node only. This mode should improve throughput for such operations, reduce delays, and allow us to restart the nodes more easily if necessary.

What’s in the lab?

We’ve been cooking up a few more enhancements for quite some time now. Here they are at a glance:

Build log service

Occasionally, build logs get corrupted. We’re fairly certain that this happens when multiple nodes update the same build log. Despite our best efforts to set these updates aside across the running build lifetime, it still happens. To tackle this issue, we’re preparing a dedicated service. Instead of updating build logs directly, all nodes will send change requests to this service, eliminating the issue and ensuring there’s only one log writer.

Central repository for all configuration files

TeamCity keeps configuration files of all the projects on disk. This location is shared with all the nodes via a network file system. However, only one node can perform direct modifications of the files on disk. Other nodes need to send a request in order to perform any modifications.

When a project is modified and its configuration files are updated, a multi-node event fires to let other nodes know they should reload this project’s settings from the disk. While this approach works most of the time, it has obvious flaws. For instance, the project may be modified further while nodes are reloading it from the disk. 

As a result, the nodes may get an inconsistent view of the project settings. We perform re-tries to work around this issue, but it’s still a workaround rather than a solid solution. This issue is probably one of the reasons why you sometimes see messages in the header saying “Critical errors in configuration files”.

To fix this problem, we want to move all of the configuration files to a Git repository. Then multi-node events will be able to send hashes of the VCS commits, which should improve the atomicity and precision of the reload operations.

In the longer term, we’ll be able to remove configuration files from the shared disk, thus making us less dependent on network storage.

Happy building!

With this version, you can now optimize your CI/CD runtime by downloading the files that your project depends on once and then reusing them within the project with the new experimental Build Cache feature.

We’re also introducing the new “Only if build status is failed” execution policy for individual steps. In addition to that, we’re also adding integration with Bitbucket Server and Data Center, several UI improvements, and accelerated artifact uploads.

Let’s take a closer look at what’s new.

Optimize your CI/CD runtime with Build Cache

We’re introducing Build Cache to the list of build features in TeamCity. Build Cache will help you optimize your CI/CD runtime by downloading the files that your build depends on once and then reusing them within the project. After an initial build run, subsequent builds will run faster as you won’t have to download the files again.

How to set up Build Cache in TeamCity

To enable Build Cache, navigate to Administration | Server Administration | Experimental Features.

Once you’ve done that, you can start configuring Build Cache on the build configuration level. To set up Build Cache rules for a specific project, navigate to Build configuration | Edit configuration | Build features | Add build feature | Build Cache (Experimental).

How does the Build Cache feature help your organization?

With Build Cache, you can significantly reduce your build time by reusing the files produced by the build. It’s especially useful if you need to download specific libraries and packages for the build to run. Using Build Cache in TeamCity will help you optimize resources and save on traffic and storage costs.

Build Cache is currently available as an experimental feature. We’re still working on improving it for our customers. Your feedback is highly appreciated – feel free to share it via the support platform.

Meet the UI improvements

In 2021, we introduced the new Sakura UI to TeamCity. We’re now working on moving towards full feature parity between the Classic and Sakura UIs.

With this release, we’re introducing a number of changes to the Sakura UI. Here’s what’s new:

Change Log tab

You can now keep track of all the changes on your TeamCity Cloud projects and configurations thanks to the Change Log tab for projects and build configurations.

Show graph option

The Show graph option is now also available in TeamCity Cloud. You can find it under the Pending Changes tab of the build configuration home page and the Changes tab of the build results page. Once you enable this feature, you’ll be able to see the changes displayed as a graph of commits to the related VCS roots.

Run steps only for failed builds

TeamCity allows you to specify the build step execution policy according to your needs. In addition to the existing options, we’ve added a new one – Only if build status is failed.

This condition allows you to add steps that are executed only when a build fails. If there are any issues, you can run tasks that roll back the changes to the remote repository with this new option.

This allows you to react quickly when a build fails and prevent deploying any broken code to production.

Integrate TeamCity with Bitbucket Server and Data Center

In addition to Bitbucket Cloud, TeamCity now integrates with Bitbucket Server and Data Center. You can now add Bitbucket Server and Data Center as a connection type, as well as create new projects and build configurations from a Bitbucket URL.

This improvement will help you streamline the development process and save time when configuring new projects.

Take advantage of accelerated artifact upload

We’re optimizing the way artifact uploads are handled in TeamCity. Starting from version 2023.03, TeamCity breaks down large artifact files into chunks and uploads these chunks in parallel. This can dramatically reduce the time it takes for the artifacts to be uploaded to the server, improving your overall CI/CD runtime.

Stay connected

We’re constantly working to improve TeamCity and introduce new features so that you can build even faster and more efficiently. Make sure to check out our public roadmap to see what’s coming next to TeamCity.

Explore the roadmap

Got a question or comment? We always appreciate your feedback, as it helps us focus on what really matters to our users. Feel free to submit your request in our support forum – we’re always glad to hear from you.

Happy building!

Other improvements are related to the integration with .NET runner and HTTPS settings. We have also updated the bundled Git and Perforce Packages in TeamCity Docker images. You can find the complete list of updates in our release notes.

As with other minor updates, TeamCity 2022.10.3 shares the same data format with all 2022.10.x releases, allowing you to easily upgrade or downgrade within these versions without the need to back up or restore.

We recommend upgrading to apply the latest improvements and security fixes to your TeamCity server.

Before you start, read our upgrade notes and use one of the following options to upgrade:

• Upgrade from your recent version with our automatic update.
• Download the new version from the JetBrains website.
• Pull the TeamCity Docker image.

Thank you for submitting issues and voting for them. Your feedback helps us improve TeamCity. If you spot any issues or have any suggestions, please report them via our forum or tracker.

Happy building!

We also continued working on TeamCity Cloud, introducing several security and maintenance improvements, enhancing integration with Amazon Web Services, and adding new configuration options.

In 2023, there are a number of features that we’re focusing on. In this blog post, we invite you to take a closer look at our development plans. Feel free to check out our full roadmap and leave your comments and suggestions.

Check out 2023 roadmap

TeamCity Q1–Q2 2023 main objectives

In addition to regular bug fixes and improvements, there are several areas that we’re focusing on this year, including a better user experience, new features for game development, and a strong focus on TeamCity Cloud.

Here are our priorities for 2023.

☁️Cloud integrations

In 2022, we saw more and more customers choose TeamCity Cloud as their CI/CD solution. The main reason for this is the ease of setting up and maintaining the cloud version. The cloud version offers a managed solution that teams don’t have to maintain themselves. All updates are rolled out automatically, and organizations always have the newest, most feature-rich version of the system.

We’re investing a lot of time and effort into developing a range of features that will let you easily set up and run builds in the cloud.

Persistent caches for cloud agents

We’re striving to help teams using TeamCity Cloud complete their builds as quickly as if they were using local installations. With persistent cloud caches, cloud build agents can transfer build dependencies to one another, saving your team time and networking costs.

Improved migration of artifacts between Amazon S3 buckets

In addition to the ability to transfer build artifacts from local storage to Amazon S3 that is already available in TeamCity, we’re working on the next improvement. With it, teams can instantly migrate artifacts between different S3 buckets, copying files within Amazon from one bucket to another. This improvement will significantly increase transfer speed, allowing teams to build even faster.

Out-of-the-box Microsoft Azure support

Running build agents in the cloud will enable teams to quickly increase the capacity of their delivery pipelines when needed. To support users adopting Microsoft Azure, we plan to bundle the TeamCity Azure plugin to make it available out of the box.

Explore all the cloud features

🌸TeamCity UI

When working with a CI/CD tool, it should feel sturdy enough so that when you try to configure pipelines, you’ll know that you won’t break anything. That’s why we’re constantly improving and polishing the TeamCity UI so that working with the CI/CD system feels like home.

The Sakura UI tab migration

The Sakura UI is now the default option in TeamCity. We’re currently working on full feature parity between the Classic and Sakura UIs.

TeamCity onboarding improvements

TeamCity offers many powerful features so that users can customize the system according to their needs. To provide better guidance and help users familiarize themselves with TeamCity, we will provide onboarding guides that will be available within the tool itself. The guides will give a better overview of all of TeamCity’s capabilities and help those users migrating from the Classic to Sakura UI.

Dark theme

We recognize how important it is for engineers to feel comfortable using the tools they work with daily. In this regard, the dark theme is one of TeamCity’s most frequently requested features. The good news is that we’ve already started working on implementing it in both the Classic and Sakura UI, and it should see the light in the near future.

Explore the roadmap

🌟TeamCity Pipelines

TeamCity Pipelines is an entirely new product experience that we’ve actively been working on. TeamCity Pipelines will provide a simple and intuitive experience for building CI/CD pipelines powered by the signature intelligence of TeamCity and JetBrains products.

The brand new visual pipeline editor in TeamCity Pipelines makes it easy to work with CI/CD pipelines of any complexity while simultaneously providing the power of TeamCity’s CI/CD engine, which can handle enterprise-level workloads. It will also bring the ability to define pipelines in YAML format.

🕹️TeamCity for game development

TeamCity is one of the most popular CI/CD solutions for game development. TeamCity integrates with all major technologies and tools used in modern game development, including Perforce. In 2023, we’re focusing on exploring how we can better support game development companies and provide support for Unity, Unreal Engine, and general CI/CD workflows in game development.

The leading companies in the game development industry are TeamCity users. Read this case study to learn more about how Playrix, one of the world’s top 3 mobile game development companies, uses TeamCity to run over 10,000 builds per day.

🏃‍♀️Build Runners and Integrations

TeamCity offers tight integrations with a number of build tools and external services. In 2023, we will continue to provide them with the best possible support. Here’s what we’re focusing on this year.

Space integration

We’ve been collaborating with the Space team to enhance the integration between TeamCity and Space. Now, we’re working on adding the ability to connect to Space right from within the TeamCity UI. We’re also developing the functionality that will make it possible to publish build statuses to the Space merge request timeline. This way, your team will always be able to access the most up-to-date information about the status of your TeamCity builds right within Space.

Bitbucket Server and Data Center connection

TeamCity already offers the ability to integrate with Bitbucket Cloud. Now, we want to expand this connection and offer the ability to add Bitbucket Server and Data Center as a connection in the Project Settings as well as to allow you to configure VCS roots via such connections.

GitHub App connection support

TeamCity has been offering GitHub support ever since its early days. This year, we’re looking into the possibility of authorizing TeamCity to work with GitHub as a GitHub App, using the GitHub API without creating separate service accounts.

👩‍💻Configuration as code

Kotlin DSL is an easy and powerful way to set up build configurations as code. This year, we keep improving this feature to give our users more flexibility in using Kotlin for their CI/CD processes.

Simplifying user experience

Using the Kotlin DSL allows experienced TeamCity users to reuse build configuration settings more naturally, saving teams time and effort. At the same time, we recognize that knowledge of Kotlin shouldn’t be an obstacle for configuring pipelines in TeamCity. That’s why we’re researching ways to simplify the user experience and make working with Kotlin DSL easier when configuring TeamCity pipelines.

Improving Kotlin DSL documentation with examples

We’re improving the Kotlin DSL API documentation. This will provide clear examples of Kotlin DSL snippets that teams can copy and paste when configuring pipelines.

Do you have any comments or suggestions? Feel free to explore our full roadmap, vote for the features in development, and request new ones. We’re always open to feedback and take all suggestions into consideration.

Explore the roadmap

We’ve created a new TeamCity plugin that brings the functionality of triggering builds by new messages into an Amazon Simple Notification Service (Amazon SNS).

Example

Intro

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

Source

Amazon SNS is one of the backbone AWS services that allows connecting various applications and services to each other. The typical use case for Amazon SNS is to provide notifications about events in one system and trigger corresponding actions in others.

We’ve set up a data processing pipeline so that you can see it in action. We’ll build a pipeline that takes a newly uploaded image, optimizes it for the web, and creates a few thumbnails of different sizes.

Our pipeline begins by placing the new image file in an S3 bucket. TeamCity is responsible for running the build configuration. The challenge is making sure that the new file is detected when it appears. The solution is simple. As with a significant part of Amazon cloud services, AWS S3 integrates with Amazon SNS out of the box with its “Publisher” role. This is a natural way to notify external systems about new files in buckets. TeamCity takes the “Subscriber” role.

The setup steps look like this:

• On the AWS side:

1. Create an AWS S3 Bucket with a name, e.g., ‘demo-storage-image-processing-pipeline’.
2. Create an AWS SNS Topic with a name, e.g., ‘new-image-into-the-demo-storage’.
3. Set the bucket to push notifications to the topic.
4. Create an IAM user for TeamCity and attach the ‘AmazonS3FullAccess’ IAM policy.
5. Create access and secret access keys.

• On the TeamCity side:

1. Create a build configuration.
2. Set up the trigger.
3. Set up an AWS Connection using the IAM user’s key.
4. Grant the build configuration permission to interact with AWS.
5. Create the pipeline steps.

Firstly, we need to create resources on the AWS side. How to create an SNS topic and S3 bucket is out of the scope of this article, but you can find step-by-step instructions in the official documentation.

Build configuration

Now it’s time to create a build configuration with the trigger that will run our processing tasks. TeamCity will ask us about a VCS root we want to use, but since we don’t need one, we can skip this.

In General Settings, specify the Artifact path. We’ll use it to demonstrate a result.

Now we have an empty build configuration. Before we start working on our steps, we need to do a couple of things:

• Extract image properties like its name and the storge bucket from the SNS Trigger 
• Grant TeamCity permission to get files from the bucket.

Trigger

When a SNS message will trigger our process, TeamCity puts the message into a parameter used in later steps. Let’s create the parameter that we will use – the value can be anything. The trigger will update the value for every running build as every new message appears.

On the Triggers page, add a new Amazon SNS Trigger. Specify its name, and don’t forget to copy the endpoint URL.

On the Build Customization tab, ask TeamCity to use the message parameter to publish an SNS message.

Don’t forget to save the trigger!

Now go to the AWS console, the SNS service, and subscribe to TeamCity by using the provided endpoint URL.

That’s it! You can check the trigger in TeamCity to see if it was updated correspondingly in the Subscription Info section:

AWS Connection

Now it’s time to grant the scripts permission to download images from the S3 bucket. On the project level, add a new connection for AWS using the same IAM user credentials we created before.

Check the connection using the Test Connection button:

Next, we want to check our build steps will work with AWS. Let’s return to our build configuration. On the Build features page using the “AWS Credentials” build feature, TeamCity will now securely provide credentials for the build steps.

Build steps

Now we are ready to set up our build steps!

In this demo, we will use the Command line runner for our build steps.

Build step #1: Get the information from the message

The custom script is:

# Extract bucket name and image name from the JSON message
BUCKET=$(echo %message% | jq -r '.Records[].s3.bucket.name')
IMAGE=$(echo %message% | jq -r '.Records[].s3.object.key')

# Ask TeamCity to remember the variables to use them 
# in the next steps
echo "##teamcity[setParameter name='env.BUCKET' value='${BUCKET}']"
echo "##teamcity[setParameter name='env.IMAGE' value='${IMAGE}']"

We will use the Docker wrapper feature to run the script inside the Docker container based on the official jq Docker image. Set ‘Run step within Docker container’ to the ‘stedolan/jq’ value. That allows us to use the jq utility on any agent, even if the agent does not have the utility preinstalled.

Build step #2: Download the image from the S3 bucket

We need to download the file from AWS S3 using AWS CLI. Again, the Docker wrapper will help us get the latest version of the AWS CLI utility.

Custom script:

aws s3 cp s3://${BUCKET}/${IMAGE} image.png

Set Run step within Docker container to the ‘amazon/aws-cli’ value.

Build step #3: Optimize image and create thumbnail

For this step, we are going to use two utilities: OptiPNG to optimize the image file size and the convert utility from the ImageMagick CLI graphical package. Unfortunately, the vendors didn’t provide us with official Docker images, so we’ll install them at run time.

Again we’ll use the Docker wrapper, but this time we use a basic ‘Ubuntu’ Docker image and install the required packages during the execution of this step. This will give us the required binaries, and we won’t litter inside the agent.

All installed packages will be erased with the container together. This approach is helpful while you construct your pipeline, but avoid it in production! When you learn all of the required dependencies, build your own Docker image with all of the packages preinstalled. It will save traffic and agent time.

Custom script:

apt -qq update &>/dev/null 
apt -qq install optipng imagemagick -y  &> /dev/null

# optimize our image
cp image.png optimized_image.png
optipng optimized_image.png

# create a thumbnail with the 200x400 size
convert optimized_image.png -strip -thumbnail 200x400 thumbnail.png

Run it!

Let’s put some .png files into our bucket using the AWS console. A few moments later, a new build will be scheduled and run. As a result, the three images are now available as build artifacts.

Of course, this is just an example; you can create much more complex pipelines. Triggers can be useful for various event-based automation tasks.

How to get it

The plugin is available in JetBrains Marketplace. Don’t hesitate to give us feedback here – we always appreciate your opinion! Our development team will be delighted to hear how you feel about the plugin.

Conclusion

With Amazon SNS integration, TeamCity can be built into multiple scenarios. It can orchestrate tasks in different cloud services based on the event trigger model. Here’s a list of AWS services that can publish messages to Amazon SNS. With the support of custom integrations, the scope of possibilities is enormous. Please share the ways you use TeamCity with Amazon SNS!