ConvertFrom-Json is not serializable

Introduction

While writing Bicep recently, I was stumped by the fact that my deployment kept failing. I spent a lot of time troubleshooting many possible ideas on what might be causing this. As JSON is involved and I am far from a JSON syntax guru, I first focused on that. Later I moved to how I use JSON in Bicep and PowerShell before finally understanding the problem was due to the fact that ConvertFrom-Json is not serializable.

Parameters with Bicep

When deploying resources in Azure with Bicep, I always need to consider who has to deliver or maintain the code and the parameters. It has to be somewhat structured, readable, and understandable. It can’t be one gigantic listing that confuses people to the point they are lost. Simplicity and ease of use rule my actions here. I know when it comes to IaC, this can be a challenge. So, when it comes to parameters, what are our options here?

  • I avoid hard-coding parameters in Bicep. It’s OK for testing while writing the code, but beyond that, it is a bad idea for maintainability.
  • You can use parameter files. That is a considerable improvement, but it has its limitations.
  • I have chosen the path of leveraging PowerShell to create and maintain parameters and pass those via objects to the main bicep file for deployment. That is a flexible and maintainable approach. Sure, it is not perfect either, but neither am I.

Regarding Bicep and PowerShell, we can also put parameters in separate files and read those to create parameters. Whether this is a good idea depends on the situation. My rule of thumb is that it is worth doing when things become easier to read and maintain while reducing the places where you have to edit your IaC files. In the case of Azure Firewall Policy Rules Collection Groups, Rules collections, and Rules, it can make sense.

Bicep and JSON files

You can read file content in Bicep using. With the json() function, you can tell Bicep that this is JSON. So far, so good. The below is perfectly fine and works. We can loop through that variable in a resource deployment.

var firewallChildRGCs = [

    json(loadTextContent('./AFW/Policies/RGSsAfwChild01.json'))

    json(loadTextContent('./AFW/Policies/RGSsAfwChild02.json'))

    json(loadTextContent('./AFW/Policies/RGSsAfwChild03.json'))

]

However, I am not entirely happy with this. While I like it in some aspects, it conflicts with my desire not needing to edit a working Bicep file once it is in use. So what do I like about it?

It keeps Bicep clean and concise and limits the looping to iterate over the Rules Collection Groups, thus avoiding the nested looping for Rules collections and Rules. Why is that? Because I can do this

@batchSize(1)

resource firewallChildPolicyWEUColGroups 'Microsoft.Network/firewallPolicies/ruleCollectionGroups@2022-07-01' = [for (childrcg, index) in firewallChildRGCs: {

  parent: firewallChildPolicyWEU

  name: childrcg.name

  dependsOn: [firewallParentPolicyWEUColGroups]

  properties: childrcg.properties

}]

As you can see, I loop through the variable and pass the JSON into the properties. That way, I create all Rule Collections and Rules without needing to do any nested looping via “helper” modules to get this done.

The drawback, however, is that the loadTextContent function in Bicep cannot use dynamic parameters or variables. As a result, the paths to the files need to be hard coded into the Bicep file. That is something we want to avoid. But until that is possible, it is a hard restriction. That is because parameters are evaluated during runtime (bicep deployment), whereas loadTextContent in Bicep happens while compiling (bicep build). So, in contrast to the early previews of Bicep, where you “transpiled” the Bicep manually, it is now done for you automatically before the deployment. You think this can work, but it does not.

PowerShell and JSON files

As mentioned above, I chose to use PowerShell to create and maintain parameters, and I want to read my JSON files there. However, it prevents me from creating large, long, and complex to maintain PowerShell objects with nested arrays. Editing these is not straightforward for everyone. On top of that, it leads to the need for nested looping in Bicep via “helper” modules. While that works, and I use it, I find it more tedious with deeply nested structures and many parameters to supply. Hence I am splitting it out into easier-to-maintain separate JSON files.

Here is what I do in PowerShell to build my array to pass via an Object parameter. First, I read the JSON filers from my folder.

$ChildFilePath = "../bicep/nested/AfwChildPoliciesAndRules/*"
$Files = Get-ChildItem -File $ChildFilePath -Include '*.json' -exclude 'DONOTUSE*.json'
$Files
$AfwChildCollectionGroupsValidate = @() # We use this with ConvertFrom-Json to validate that the JSON file is OK, but cannot use this to pass as a param to Bicep
    $AfwChildCollectionGroups = @()
    Foreach ($File in $Files) {
    try{
        $AfwChildCollectionGroupsValidate += (Get-Content $File.FullName -Raw) | ConvertFrom-Json
        # DO NOT PUT JSON in here - the PSCustomObject is not serializable and passing this param to Bicep will than be empty!
        $AfwChildCollectionGroups += (Get-Content $File.FullName -Raw) # A string is serializable!
        }
        Catch
        {
        write-host -ForegroundColor Red "ConvertFrom-Json threw and error. Check your JSON in the RCG/RC/R files"
        Exit
        }
    }

I can then use this to roll out the resources, as in the below example.

// Roll out the child Rule Collection Group(s)

var ChildRCGs = [for (rulecol, index) in firewallChildpolicy.RuleCollectionGroups: {

  name: json(rulecol).name

  properties: json(rulecol).properties

}]

Initially, the idea was that by using ConvertFrom-Json I would pass the JSON to Bicep as a parameter directly.

$AfwChildCollectionGroups += (Get-Content $File.FullName -Raw) | ConvertFrom-Json

So not only would I not need to load the files in Bicep with a hard-coded path, I would also not need to use json() function in Bicep.

// Roll out the child Rule Collection Group(s)
var ChildRCGs = [for (rulecol, index) in firewallChildpolicy.RuleCollectionGroups: {
  name: rulecol.name
  properties: rulecol.properties
}]

However, this failed on me time and time again with properties not being found and what not. Below is an example of such an error.

Line |
  30 |          New-AzResourceGroupDeployment @params -DeploymentDebugLogLeve …
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 2:46:20 PM - Error: Code=InvalidTemplate; Message=Deployment template validation failed: 'The template variable 'ChildRCGs' is not valid: The language expression property 'name' doesn't exist, available properties are ''.. Please see    
     | https://aka.ms/arm-functions for usage details.'.

It did not make sense at all. That was until a dev buddy asked if the object was serializable at all. And guess what? ConvertFrom-Json creates a PSCustomObject that is NOT serializable.

You can check this quickly yourself.

((Get-Content $File.FullName -Raw) | ConvertFrom-Json).gettype().IsSerializable

Will print False

While

((Get-Content $File.FullName -Raw)).gettype().IsSerializable

Will print True

With some more testing and the use of outputs, I can even visualize that the parameter remained empty! The array contains three empty {} where I expected the JSON.

ConvertFrom-Json is not serializable

I usually do not have any issues with this in my pure PowerShell scripting. But here, I pass the object from PowerShell to Bicep, and guess what? For that to work, it has to be serializable. Now, when I do this, there are no warnings or errors. It just seems to work until you use the parameter and get errors that, at first, I did not understand. But the root cause is that in Bicep, the parameter remained empty. Needless to say, I wasted many hours trying to fix this before I finally understood the root cause!

As you can see in the code, I still use ConvertFrom-Json to test if my JSON files contain any errors, but I do not pass that JSON to Bicep as that will not work. So instead, I pass the string and still use the json() function in Bicep.

Hence, this blog post is to help others not make the mistake I made. It will also help me remember ConvertFrom-Json is not serializable.

Project Bicep, an ARM Domain-Specific Language

Project Bicep

Project Bicep? Biceps? Do you mean like bicep curls? Muscles? What does this have to do with ARM or ARM templates? Well, to master ARM templates, we can use a little extra power. So, It’s a joke so bad it’s good as Microsoft’s Alex Frankel put it.

Impressive power but not the kind of biceps we are talking about (image by Eduardo Romero on Pexels.com)

Over the years, I have noticed a couple of challenges when it comes to Infrastructure as Code (IaC). It is not an easy thing to achieve in practice. Not only in the cloud but anywhere. It is a significant hurdle in achieving IaC. Maybe you have the same experiences.

Azure Infrastructure as Code

In Azure, one of the biggest challenges has been the learning curve when it comes to writing the JSON. JSON, the “human-readable” data interchange format that brings ARM and ARM templates to live. It isn’t something you pick up super quickly and turns out to be harder and harder to use when things become more complex and diverse.

Other challenges are related to managing the templates, getting pipelines set up reliably and consistently for all resources in Azure tenants, subscriptions, resource groups, etc. It is not something that I would call inviting and easy to do.

Then there are the real-world realities we need to handle. There is a ton of “stuff” out there where deployment, configuration, orchestration, and change happens in different ways. How does one onboard all that in an IaC process without too much risk of breaking things? Unfortunately, this is tedious and fragile.

We like Infrastructure as Code

For many people, the above is a bit discouraging. Don’t get me wrong. People see, understand, and like the idea of Infrastructure as Code. They just have a hard time getting there. There are all sorts of tools for various environments and needs. We have all at least heard and probably looked at Chef, Ansible, Puppet, or Terraform. There are many others still, but I just listed the ones that have been getting some serious attention over the last four years. Choosing one is losing the benefits of another. Using them all is an operational, skills, and management challenge. They all have their strengths and weaknesses. The main differences are whether they take a procedural, declarative, or an orchestral approach to getting the job done.

While orchestration is very popular, it does feel a bit like a failure, but that is “emotion”. Why? Well, because in the end, we cannot manage change very well and end up throwing everything away and replacing it with a new deployment that has the changes in there. Everything is a cow now that gets slaughtered and replaced when it doesn’t function as expected or needs to change. That works well for lightweight and fast implementations. It is somewhat painful when using this in more massive deployments. But still, when looking at the results and preventing configuration drift, it gets the job done.

But even the best tools have issues that can be best described as “death by a thousand cuts.” The concept is simple, but that doesn’t make it easy to do!

Microsoft has heard this feedback

We like Infrastructure as Code. We do find it too hard to do well, especially if that is not the bulk of your work, and you are not a guru at it like Stanislav Zhelyazkov.

When Microsoft asked on Twitter, “What do you think is a knowledge gap for traditional #ITPros when it comes to transitioning to the cloud” I replied, “The biggest skills hurdle is related to IaC. ARM is tedious and hard to learn for many, yet a cornerstone … Fix that, and we can move 10 faster in any cloud journey.”.

Project Bicep
My honest reply to Microsoft’s Anna Chu

The above is not new feedback, far from. But recently (Build 2020) they talked publically about what they are doing about it if I recall correctly. Yes, Microsoft is addressing this challenge. Just last week I saw Project Bicep go public on Github!

So what is project Bicep?

Bicep is a project Microsoft announced at Build 2020 (May). It delivers what Microsoft calls Transparent Abstraction over Azure ARM and ARM templates.  ARM ==> Bicep get the joke? Ok, never mind, it is terrible to search for, however. You get a lot of irrelevant hits.

It has a couple of goals as you can learn from the video.

  • Human friendly, so readability and comprehensibility are essential. You have to be able to understand what you read and write without much effort.
  • What you write will create or compile JSON for you. Microsoft now seems to like “Transpiles” for this. Where earlier the sort of made the analogy of JSON being some sort of the Intermediate Language (IL) of IaC.
  • If you think of JSON of an IL (as MSFT suggests), it is easy to see that, just like with .NET, you might see different languages use to achieve the same goal. But for now, that is not the goal. The goal is to get a working, functional declarative language that is suitable for all kinds of users. We’ll see where this ends up.
  • It focuses on modularity, so no, it will not create giant ARM templates, but modular ones. That means there is multi-file support.
  • It should evolve at the speed of Azure, so no waiting for six months to get new functionality implemented. Microsoft calls this “transparent abstraction.”
  • They plan a migration/conversion/export tool for existing ARM JSON!

Read up on project Bicep over here https://github.com/Azure/bicep. It clarifies the current state of what Bicep is and is not. I hope this moves fast and delivers better tooling to make Infrastructure as Code a better, more accessible, and more achievable goal for all of us.

WARNING

Bicep is at a super early stage of its existence. This is the earliest Alpha you can imagine. It is going to break, barf, and probably puke on your Azure stuff once in a while. So please, DO NOT USE THIS IN PRODUCTION. Right now it is only to get a feel for it, tinker around and get some feedback. This is you only and final warning.

In all honesty, it is very raw and as a (non-Linux) hardcore dev, this is not love at first sight for me, as I had hoped to use PowerShell for this. I hope it will mature and I will grow to love it and like using it much more than ARM. Anyway, dive into the bicep tutorial to see what you think.