Contents tagged with Workflow Manager
This is the first “real” posts in the Workflow Manager Disaster Recovery series. In this post I will show you what you need to do to prepare for Disaster Recovery (DR) situations when working with Workflow Manager and Service Bus.
Let’s start with the obvious pieces. You should run your Workflow Manager farm on three (3) servers (for more on this discussion see the SPC356 session). Running on three servers are important not just for high-availability it might save you from going into DR mode. DR should be considered as the last resort. You should also consider one or more SQL Server high-availability options, more on this later.
Data tier preparations
The Data tier in the Workflow Manager (WFM) /Service Bus (SB) is the actual SQL Server databases deployed when setting it up. It’s a minimum of six (6) databases – three for WFM and three for SB. These databases should of course be backed up – all DR options for Workflow Manager requires that you have a database backup/replica to restore from. Configure a “good” backup schema with full, differential and transaction log backups according to your SLA. You should also keep these backups in “sync” – that is don’t do SB backups on odd weeks and WFM backups even weeks for instance. They are normally not that large databases and the backup process should be pretty fast.
Secondly you must think of the high-availability (HA) options for these databases. As I said previously, this can save you from going into DR mode. Most SQL HA options can be used with Workflow Manager and Service Bus, for instance:
- Always On Availability Groups with sync commit
If you need to shorten your DR time/RTO (Recovery Time Objective) then you should also not only rely on SQL backups but implement some replication technique with SQL to replicate data to a secondary location. The following techniques works great:
- Log Shipping
- Always On Availability Groups with async commit
Compute tier preparations
Once you are in control of the databases, backups and optionally replicas you could also consider weather you would like to have a cold or warm standby. A warm standby will shorten your RTO compared to a cold standby. A hot stand by Workflow Manager farm is NOT supported (a hot standby is a secondary running instance).
- Cold standby – when a disaster occurs you create a brand new Workflow Manager/Service Bus farm using scripts and restore data from backups or replicated data.
- Warm standby – a secondary farm configured but all the nodes (WFM and SB) are turned off. Use scripts to resume the nodes. Once the nodes are resumed you need to run the consistency verification cmdlet (Invoke-WFConsistencyVerifier, more on this later in the series).
The Symmetric Key is the key
In order to be successful or succeed at all you must keep track of the Symmetric key of the Service Bus. Without the Symmetric Key you cannot restore the Service Bus and you will loose all of your data. Keep it in a safe place!
You can find the Symmetric Key by running the following PowerShell command:
Get-SBNamespace -Name WorkflowDefaultNamespace
You’ve just read the first post of this series and you’ve learnt the basics in how to prepare for, and possibly avoid, a Disaster Recovery situation for Workflow Manager and Service Bus. In the following posts we will discuss what do do if and when the disaster happens…
Welcome to a new series of blog posts in which we will focus on the Disaster and Recovery (DR) routines for Workflow Manager 1.0 in combination with SharePoint 2013. During SharePoint Conference 2013 me and SharePoint sensei Spencer Harbar presented a session called “Designing, deploying, and managing Workflow Manager farms” (watch the video recording). During that session we discussed different DR options for Workflow Manager and the Service Bus and we got tons of questions on that specific topic. We did not have time to go into details and we did not show any of the necessary scripts/routines you need to do when restoring a Workflow Farm or Workflow Scopes, and there is very little information available on that topic on the interwebs – so that is why this new blog series is being posted.
Index of posts
This blog post is just the introductory post and will be used as a place holder for all the posts in the series. Instead of writing one behemoth post I will split them into multiple ones and that will be a little bit easier for you to consume as well.
These are the planned/written/unwritten posts and will be linked to when posted.
- Workflow Manager Disaster Recovery – Index post (this post)
- Workflow Manager Disaster Recovery – Preparations
- Workflow Manager Disaster Recovery – Recover a single Scope
- Workflow Manager Disaster Recovery – Recover a single database
- Workflow Manager Disaster Recovery – Recover a single machine
- Workflow Manager Disaster Recovery – Recover a full farm
If you have any ideas on what more options to cover then feel free to post a comment below.
Note:All examples in this series uses Windows Server 2012 R2, SharePoint 2013 Service Pack 1 and Workflow Manager 1.0 Refresh (+ Service Bus 1.1).
For proper configuration of the Workflow Manager Farm see the following posts by Spencer Harbar:
- Workflow Manager Farms for SharePoint 2013 Part One: Core Concepts, High Availability, Certificate and SharePoint considerations
- Workflow Manager Farms for SharePoint 2013 Part Two: End to End Configuration using Auto Generated Certificates and NLB
- Workflow Manager Farms for SharePoint 2013 Part Three: Switching an existing farm to use Domain CA issued certificates
- Workflow Manager Farms for SharePoint 2013 Part Four: End to End Configuration using Domain CA issued Certificates
When using the Web Platform Installer to download and/or install Workflow Manager you can no longer download and install Workflow Manager 1.0 and Workflow Manager 1.0 CU1. The only option is to download Workflow Manager 1.0 Refresh (which essentially is CU2). So when installing a new Workflow Manager farm for SharePoint or just because you want to rock some workflows you have to use Workflow Manager (WFM) 1.0 Refresh. Unless you’ve been smart and previously downloaded and saved the original Workflow Manager. When using WFM 1.0 Refresh you also need to download Service Bus 1.1.
Trouble in paradise!
Installing the bits works like a charm and if you configure the Workflow Manager farm and the Service Bus using the wizard everything is golden. But if you’re a real professional then you of course use a scripted and repeatable approach and use PowerShell to build out your Workflow Manager farm.
Most of the PowerShell commands and steps works as expected when creating the Service Bus farm, the Workflow Manager Farm, adding the first Service Bus host and creating the Service Bus namespace. But when you try to add a Workflow Host to the machine, using the Add-WFHost cmdlet, and when it tries to connect to the Service Bus namespace you get an error like this:
Add-WFHost : Cannot validate argument on parameter 'SBClientConfiguration'. Could not load file or assembly 'Microsoft.ServiceBus, Version=126.96.36.199, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
If we crack up the Microsoft.Workflow.Deployment.Commands.dll using our favorite tool of choice of disassembly we can see that it actually references version 188.8.131.52 of Microsoft.ServiceBus.dll. Obviously a bug in this release!
Solve the problem using basic .NET features
The easiest way to work around it is of course to use the wizard, which works, but that is for slackers. So let’s instead use some basic .NET framework knowledge and make an assembly redirection. Assembly redirection is a way to redirect one version of an assembly to another, so that when the .NET framework tries to load version 184.108.40.206 we ask it to load version 220.127.116.11 (which is the Service Bus 1.1 version number). There is a couple of ways to do this;
- We can update the machine.config file and make a machine wide redirection – I do not recommend this approach for obvious reasons
- We can update the PowerShell.exe.config file and make the redirection for all PowerShell sessions – not a good approach either
- Or we can dynamically load the configuration data in our PowerShell session – Heureka, that’s the way to do it!
In order to do the assembly redirection we need to create a .config file which tells the .NET framework how to do the redirection. This is how the .config file should look like, and in this case I named it wfm.config. If you want do dig deeper into Assembly Redirection there is of course documentation on MSDN.
<?xml version="1.0" encoding="utf-8" ?> <configuration> <runtime> <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> <dependentAssembly> <assemblyIdentity name="Microsoft.ServiceBus" publicKeyToken="31bf3856ad364e35" culture="en-us" /> <bindingRedirect oldVersion="18.104.22.168" newVersion="22.214.171.124" /> </dependentAssembly> </assemblyBinding> </runtime> </configuration>
To load this configuration into the current AppDomain we use the SetData() method of the current AppDomain and pass in the path to the file like this:
$filename = Resolve-Path .\wfm.config [System.AppDomain]::CurrentDomain.SetData("APP_CONFIG_FILE", $filename.Path)
As you can see from above, we create a variable with the path to the .config file and then grab the current AppDomain and invoke the SetData() method with the APP_CONFIG_FILE key and the filename as value.
Once you have created the file and executed those two lines of PowerShell you can use the Add-WFHost once again and continue your Workflow Manager configuration.
In this post you’ve seen a (major) issue with the Workflow Manager 1.0 Refresh, where you cannot add new Workflow Hosts to your Workflow Manager farm, due to an invalid assembly reference. Using “basic” .NET framework knowledge we can “fix” this issue by using assembly redirection. Hopefully someone from the WFM team will pick this up and make a refresh of the refresh.
SharePoint 2013 introduces a whole set of new and improved features. One of the things that is both new and vastly improved is the Workflow platform. Workflow is no longer a part of the SharePoint core infrastructure, but instead a separate server product. Even though ye olde Workflow platform, 2010 style, is still in the product for backwards compatibility. SharePoint 2013 leverages the Azure service called Workflow Manager 1.0. (Not the cloud version but a local on-premises installation).
The Workflow Manager 1.0 server application should be installed on a separate set of servers – not on the SharePoint servers (remember; just because you can doesn’t mean you should). This introduces for some organizations more investments in physical and virtual hardware. So, what happens if you have several farms? Do you need several Workflow farms? That can be expensive! Well, as always in the SharePoint world - It depends! If you come to the conclusion that you could use the same Workflow Farm and hardware for managing all your SharePoint Farms workflows then you can actually use the same Workflow Manager 1.0 farm. Usual disclaimers apply here – it’s all up to you and your organization to decide upon designing your workflow infrastructure for this scenario, remember this can make backup/restore and DR a bit problematic since now your farms are sharing the same workflow databases etc.
Let me show you a multi-tenant Workflow farm, how it works and how you do to set it up properly…
Multi-tenancy in Workflow Manager 1.0
Azure Workflow Manager 1.0 is multi-tenant aware through Scopes. “A scope is a named and securable container for Activities, Workflows, Instances, configuration and child Scopes.” – that is directly stolen from the Workflow Manager documentation. Everything that runs inside Workflow manager belongs to a Scope. By default a Root Scope is created when you set up the Workflow Manager farm. You can see details about the Root Scope by navigating to the HTTP/HTTPS port of your farm. The URL should look something like this: http://wffarm.corp.local:12291 for HTTP and https://wffarm.corp.local:12290 for HTTPS.
Connecting SharePoint 2013 to the Workflow Manager 1.0 farm
I’m not going to dive in to how to configure the Workflow Manager farm; it’s a wizard or PowerShell thingie and it’s very well documented – you can’t fail! Once you have the Workflow farm up and running you MUST install the Workflow Client 1.0 components on all the Web Server in your SharePoint 2013 farm (you can get the bits using the Web Platform Installer).
All you have to do then is to register the Workflow Manager 1.0 farm with the SharePoint 2013 farm using the Register-SPWorkflowService cmdlet. This will register the SharePoint 2013 farm with the Workflow Manager and vice versa and it will also create the Workflow Service Application Proxy in SharePoint.
You execute the cmdlet like below and pass in a URL to a site in your SharePoint 2013 farm (any site will do) and the URL to your Workflow Manager farm (if you’re using HTTP you also need to add the AllowOAuthHttp parameter).
Register-SPWorkflowService -SPSite https://teams.corp.local -WorkflowHostUri https://wffarm.corp.local:12290
Not only will this cmdlet do the things mentioned above, it will also create a new Scope in the Workflow Manager farm (under the Root Scope) called SharePoint. You can verify this by navigating to Workflow Manager URL and append /SharePoint, like this: https://wffarm.corp.local:12290/SharePoint. In the XML returned you can see the URL to the SharePoint 2013 metadata endpoint (https://teams.corp.local/_layouts/15/metadata/json/1) under the SecurityConfigurations element. Now, that was one SharePoint 2013 farm connected to the Workflow Manager farm.
So, let’s jump on over to our second SharePoint 2013 farm, that we now decided should use the same Workflow farm! If we run the same cmdlet in that farm (using a different SPSite parameter URL of course) we get a nice error that says “An existing scope named “SharePoint” already exists in the workflow server”.
You see, this scope is already taken by some other farm (the one we previously connected)! The clever one might think that oh, I’ll just use the Force parameter of that cmdlet and override this. Well, if you’re that smart then you will hijack that Scope from the other farm. The Scope will be recreated and the other farm will have a non-functional Workflow Service (the security configuration will now point to the SharePoint farm that hijacked the Scope).
Note: the Force flag might be useful if you really want to re-create the Workflow connection using the same Scope name.
Instead you should use the ScopeName parameter of the Register-SPWorkflowService cmdlet. That parameter will create a new Scope in the Workflow farm, and with that create an isolated container for this new SharePoint farm. So on our second farm we run this PowerShell cmdlet:
Register-SPWorkflowService -SPSite https://farmb.corp.local -WorkflowHostUri https://wffarm.corp.local:12290 -ScopeName FarmB
Once this command is issued your Workflow Manager farm will have two child Scopes under the Root Scope; SharePoint (the first farm we connected) and FarmB (the second one with the ScopeName parameter). You can verify the second scope by browsing the the Workflow farm and append the Scope name to the URL, like this: https://wffarm.corp.local:12290/FarmB.
Now you have two SharePoint 2013 farms using the same Workflow Manger 1.0 farm, and each SharePoint farms workflows are isolated using the Scopes in Workflow Manager. Cool!
I’ve just shown you how easy it actually is to share a Workflow Manager 1.0 farm between several instances of SharePoint 2013 farms. Now it’s up to you to decide on weather this is an appropriate way for your organization to build your infrastructure or not. Remember to plan your resources and your DR strategy!