Suggestion: Multi document output

[pacorreia]

First of all awesome work with this module!

It really helped me a lot, but I had to find a workaround for saving multi documents.

My use case is very simple, a K8s manifest with several resources defined.

I Import an existing yaml file, do my changes and by the time I have to save it to a new yaml file this is what I use:

function Save-ArrayAsYamlFile {
<#
.SYNOPSIS
    Saves an array of objects as a YAML file.

.PARAMETER InputObject
    The array of objects to save as a YAML file.

.PARAMETER OutputPath
    The path to save the YAML file.

.EXAMPLE
    Save-ArrayAsYamlFile -InputObject $array -OutputPath "path/to/output.yaml"
#>
  param(
    [Parameter(Mandatory = $true, Position = 0, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true, HelpMessage = "The array of objects to save as a YAML file.")]
    [array]$InputObject,

    [Parameter(Mandatory = $true, Position = 1, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true, HelpMessage = "The path to save the YAML file.")]
    [string] $OutputPath
  )

  $ErrorActionPreference = "Stop"

  try {
    Import-YamlModule -Force -NoClobber

    Write-Debug "Document count: $($InputObject.Count)"
    Write-Debug "Document array: $InputObject"

    for ($i = 0; $i -lt $InputObject.Count; $i++) {
      if ($i -eq 0) {
        Write-Debug ("Writing the first document to '$OutputPath' as:`n{0}`n" -f ($InputObject[$i] | ConvertTo-Yaml) )

        $InputObject[$i] | ConvertTo-Yaml | Out-File -Path $OutputPath -Force

        Write-Debug "Content saved."
      }
      else {
        Write-Debug ("Appending the next document to '$OutputPath' as:`n{0}`n" -f ($InputObject[$i] | ConvertTo-Yaml) )

        "---" | Out-File -Path $OutputPath -Append

        $InputObject[$i] | ConvertTo-Yaml | Out-File -Path $OutputPath -Append

        Write-debug "Content appended."
      }
    }
    Write-Debug "Updated deployment manifest has been written to '$OutputPath'."
  }
  catch {
    Write-Error -Message "Error in script $($MyInvocation.MyCommand.Name) at line $($MyInvocation.ScriptLineNumber): $($_.Exception.Message)"
    throw $_
  }
}

Found that when converting from YAML, I get an object of type System.Array, indexed by [int]

So was easy for me to be lazy and generate the YAML for each item at a time, add the "---" at the end, save to the file and keep going

Hope it can serve as base for a much more elegant solution, for now, this one just works like charm for me

[gabriel-samfira]

Hi @pacorreia

The issue of outputting multiple documents has been brought up before and the decision back then was to wait and see.

The current implementation of ConvertTo-Yaml allows you to pipe objects to the commandlet. When you do this, the input that comes in through the pipe is iterated over. So if you pipe an array, an action is taken for each object in the array.

We work around this by using the BEGIN, PROCESS and END blocks of an "advanced" powershell commandlet. But a simple array is indistinguishable from an array of documents. So the result you get will probably not look like anything you would expect.

We can add a new commandlet as you suggested, but I believe we could potentially have a cleaner solution to this. We can define a new custom type like PowershellYamlMultiDocument (I am terrible at naming things), which would represent an array of objects that would be serialized as a single document.

We could then have ConvertTo-Yaml recognize that object as a collection of objects that need to be outputted as a single multi-document file. Some checks would need to be done in the END block of the commandlet to make sure we don't pipe in a mix of PowershellYamlMultiDocument and non PowershellYamlMultiDocument objects, and then output the result.

This would allow us to still use ConvertTo-Yaml as the sole commandlet for serializing to yaml.

What do you think?

[pacorreia]

Hi @pacorreia

The issue of outputting multiple documents has been brought up before and the decision back then was to wait and see.

The current implementation of ConvertTo-Yaml allows you to pipe objects to the commandlet. When you do this, the input that comes in through the pipe is iterated over. So if you pipe an array, an action is taken for each object in the array.

We work around this by using the BEGIN, PROCESS and END blocks of an "advanced" powershell commandlet. But a simple array is indistinguishable from an array of documents. So the result you get will probably not look like anything you would expect.

We can add a new commandlet as you suggested, but I believe we could potentially have a cleaner solution to this. We can define a new custom type like PowershellYamlMultiDocument (I am terrible at naming things), which would represent an array of objects that would be serialized as a single document.

We could then have ConvertTo-Yaml recognize that object as a collection of objects that need to be outputted as a single multi-document file. Some checks would need to be done in the END block of the commandlet to make sure we don't pipe in a mix of PowershellYamlMultiDocument and non PowershellYamlMultiDocument objects, and then output the result.

This would allow us to still use ConvertTo-Yaml as the sole commandlet for serializing to yaml.

What do you think?

Hey,

Got you perfectly, and knew it was about the type, indeed is hard to distinguish where to draw a 'line' between each documento.

Your suggestion looks sensible to me and would work.

[gabriel-samfira]

I'll try to create a proof of concept soon. And we can discuss on the PR.

[pacorreia]

Reach out if you feel like discussing it before the PR

[skaravos]

Just to throw another idea out there for how this might be implemented:

What about defining a YamlStartDocument sentinel class inside the module, that when piped into the ConvertTo-Yaml function would instruct the cmdlet to insert the document separator --- before continuing on with additional objects?

Users of the module would access the sentinel through a Get-YamlStartDocSentinel function:

# user has an arbitrary collection of objects they want to write as separate documents
> $userCollection = @(
  [ordered]@{ a=1; b=2; c=3 } 
  ,@(4,5,6)
  [ordered]@{ d=7; e=8 }
)

# they can inject a StartDocSentinel between each element in the pipeline
> $userCollection | % { Get-YamlStartDocSentinel; $_ } | ConvertTo-Yaml
---
a: 1
b: 2
c: 3
---
- 4
- 5
- 6
---
d: 7
e: 8

The module could even assign a clever alias for the sentinel function like: yaml---

> $userCollection | % { yaml---; $_ } | ConvertTo-Yaml
---
a: 1
b: 2
c: 3
---
- 4
- 5
- 6
---
d: 7
e: 8