Today we are happy to announce the 1.1 release of Backyards (now Cisco Service Mesh Manager), Banzai Cloud's automated and operationalized service mesh product built on Istio. This is an announcement post describing the new features of Backyards (now Cisco Service Mesh Manager). If you're not familiar with Backyards (now Cisco Service Mesh Manager) yet, and want to know why we decided to build this product, we suggest reading the blog post about the first major release.

Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.

## TL;DR In Backyards 1.1 we've added the following major features:

• Multi-cluster meshes can be managed from the CLI: use backyards istio cluster attach to add a new cluster to a mesh.
• Outbound traffic from the Envoy proxies can be restricted through the Sidecar resource to reduce resource consumption
• The traffic tap feature helps debugging HTTP traffic that flows through an Envoy sidecar proxy
• mTLS settings can be configured for individual namespaces or services through the dashboard or the CLI
• Traffic to external services can be monitored through the dashboard's topology view
• TCP throughput metrics are now properly displayed on the dashboard
• The Overview page shows a summary of the service mesh with global metrics, global settings, and validations
• Multiple ingress and egress gateways can be added through Kubernetes custom resources

What's new

We have been working hard over the previous two and a half months to push as many things into this release as possible, and the length of the changelog shows it. Previously we had already published previews of two major features in this release: support for sidecar resources, and building multi cluster service meshes from the CLI. We'll recap these first, then we'll continue with the remaining major features.

Multi-cluster meshes from the CLI

We noticed increased interest in hybrid- and multi-cloud solutions at most of the companies we had discussions with. Use-cases can range from scaling out to the public cloud for peak workloads, through cost effectiveness to cloud migration. But regardless of the reason, many installations need to be able to span across multiple clusters. The service mesh can be a good fit for some of these use-cases by connecting clusters and handling the complexity of inter-cluster communication. Backyards - and the underlying Banzai Cloud Istio operator - have been built with these scenarios in mind from day one. Backyards (now Cisco Service Mesh Manager) collects and displays telemetry data properly from these clusters, and makes routing and policy configuration as easy as in a single cluster. There had been one major feature still missing: the ability to manage (attach, detach, display) clusters easily using Backyards. From the 1.1 release, the CLI can be used to build a multi-cluster mesh with two commands: a backyards install on one cluster to install the control plane, and a backyards attach with the Kubeconfig of the second cluster to add in. We've already published a blog post about this topic, check it out if you want to read the details.

To realize the full power of the Banzai Cloud platform, let Pipeline manage the clusters, and use Backyards to manage the service mesh layer on top. Read more about this here

Fine tune sidecar proxy configuration

Istio's Envoy proxies can consume a lot of memory if not configured properly. By default Pilot automatically pushes the same configuration to all proxies in the mesh. That configuration contains information about every workload in the mesh. Istio's Sidecar resource can be used to fine tune the configuration received by the sidecar proxies. For example, you can restrict the set of services that the proxy can reach to specific workloads or namespaces. It means a smaller configuration to handle for Envoy, and therefore reduced memory usage. From the 1.1 release, Backyards can be used to easily set up outbound traffic restrictions using the dashboard or the CLI. It even offers recommendations by monitoring and analyzing the current traffic between services. To learn more about the sidecar resource, read the corresponding blog post.

Traffic tap

It is (in)famously hard to debug connection errors in Istio when something goes wrong. One of the main value propositions of Backyards is to help diagnosing and debugging issues within the service mesh. The new traffic tap feature is a further step in this direction. This feature brings you an option to easily trace all the HTTP requests that flow through a specific Envoy proxy. Just select a workload from the dashboard, and you'll see the inbound and outbound requests from that proxy. This can help you gain some insight into whether your requests are heading to the right destination, or what the failure cause might be. We'll write a post about the tap feature soon, stay tuned for the details and examples.

Advanced mTLS policy settings

One of the exciting new features of Istio 1.4 was automatic mutual TLS support, which brought some long-awaited convenience to Istio users configuring mTLS for their applications. In previous Istio releases, it was necessary to create Destination Rules to use mTLS. The new automatic mTLS feature allows you to adopt mutual TLS without creating any Destination Rules. The Banzai Cloud Istio operator - underlying Backyards - have had this feature configurable since the 1.4 release. But with the new release, Backyards further simplifies the mTLS concept by providing easy to understand CLI commands, and an intuitive UI to configure mutual TLS for specific services, namespaces, or for the whole mesh. On one hand it makes it easier to decide if a specific service is using mTLS or not. It's not always that as easy as it seems: there are MeshPolicies, namespace-wide and workload-wide Policies, and who knows without looking at the docs if the CR is valid in a different namespace or not. On the other hand, Backyards makes it easy to configure mTLS. No more YAML syntax errors, hard to debug typos, or mis-specified namespaces.

Displaying external services

An Istio service mesh has a few different ways of reaching services that are external to the mesh. External services are everything that are not defined in Istio’s internal service registry. The default setting allows all outbound traffic through an Envoy Passthrough listener. In a production environment it is recommended to only allow outbound traffic that's registered with Istio through Service Entries. Otherwise traffic is trapped with an Envoy BlackHole listener. The above scenarios are now properly displayed in the Backyards (now Cisco Service Mesh Manager) topology view, external traffic is shown either through Passthrough, Blackhole or Service Entry nodes. Outbound traffic policy mode is configurable from the CLI using the backyards istio outbound-traffic-policy command. In this release, Service Entries are display only, configuration through Backyards is on our roadmap.

Handle TCP metrics

In previous Backyards (now Cisco Service Mesh Manager) releases, HTTP and TCP traffic had not been distinguished properly. Simple TCP traffic was visible on the topology view, but throughput metrics were not present. In this release we've made a lot of improvements around how TCP traffic is handled, so it's corresponding metrics - like throughput (MiB/s) - are now accurately displayed on the graph view of the Backyards UI.

Overview page

The overview page is the best place to get a summary of the current state of your service mesh. It displays some global statistics, configs and metrics of the mesh, like the overall error rate and latency values, or the number of pods and services available. Mesh-wide mTLS configs can also be changed here, and you'll also see the current status of the attached clusters in a multicluster setup. Last but not least, the overview page shows all validation warnings and errors of the service mesh configuration.

Manage multiple ingress and egress gateways

It is a common use-case to have multiple ingress gateways in an environment (we also have one for Backyards), and we've had a feature request about it for a long time. This is more of a Banzai Cloud Istio operator feature (for now), but the new release of Backyards brings a new version of the underlying operator as well. And because we suggest using Backyards to install the Istio operator and therefore Istio as well, we'll highlight this addition here. The Istio operator got a new CRD, called the MeshGateway that can be used to add and configure a new Istio ingress or egress gateway to the mesh. When you create a new MeshGateway CR, the operator will take care of configuring and reconciling the necessary resources, including the Envoy deployment and its related service. We'll have a blog post about this feature soon, stay tuned if you're interested in the details.

Bugfixes

Along with the new features, we also did some "battle hardening" of Backyards. Some of the larger, notable bug fixes include:

• better handling of workloads with custom controllers
• mirrored HTTP traffic is now properly displayed
• fixed some latency metric values on the topology view
• fixed error rates in the workloads list view
• handling cases when owners are missing for workloads

Wrap-up

While this release contains a ton of new features, we won't stop on our journey to bring you the best service mesh experience. Our next release will focus on ingress and egress gateway management, stay tuned!

Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.