profile image

Kevin LaBranche

Teacher @


I’ve been a part of an Architect Forum that Clear Measure (@ClearMeasure) and Jeffrey Palermo (@Jeffreypalermo) have been hosting monthly.  With many conversations over these forums Jeffrey asked me to join him on his podcast.  I am honored to join the multitude of amazing alums on the podcast and definitely feel inadequate.  However, I won’t let that stop me and don’t let that stop you.  Show up, give your best, it will help someone and help you grow as well.  We all have something to share.  While we conversed about DevOps and build scripts the underlying thread is how I approach my architecture position with mentorship as a very strong and necessary component to success.

Variable groups allow for sharing of configuration values that can be shared by all pipelines and releases in the same project.   

I have authored and maintained a few hundred Azure DevOps pipelines and releases over the last 3-5 years.  One of the conventions that I developed is around the use of variable groups. 

The conventions I have set in our team provide for quick recognition of where a value is sourced from. The magic of variable groups matching on their name to replace a value in a configuration for a stage is too magical.  It’s super convenient until there’s a mismatch.  Not all tokenize replacement tasks are created equal either.  Some can be configured to error with mismatches, some can’t.  I have found this can easily be a source of wasted effort when troubleshooting an issue and/or worse the green release but a configuration value didn’t get replaced. 

Conventions

1. Add a prefix of your choosing to the name of each variable in the group. 

2. Explicitly map variables in your pipeline to them.

3. Use Release scope for variable groups mapped to stages.

4. How to override a variable group value.

5. Don’t lock a variable (make it a secret, pad lock icon) if it really doesn’t need to be hidden including logs.

6. If you have to lock a value consider instead using Azure Key Vault and link it to a variable group.

7. Create a developer wiki for the variable groups.

Add a prefix

Adding a prefix like vg-variableName quickly allows a team member to know where this value is sourced from. If it’s just a value directly sourced in the pipeline variables we don’t add a prefix.

Bonus – If you have infrastructure based configurations such as what to name an IIS application, I recommend you prefix that as well. 

The prefixes or lack thereof is information that the team can use to more quickly assess pipeline variables.

Explicitly map

By explicitly mapping our variable’s used in a variable group in our pipeline variables we remove the magic in replacing variables by name match for groups.  At a glance we know if variables are coming from a group and which one’s in the group we are using.  While this means some initial extra setup, it pays long term dividends in maintenance and troubleshooting.

variable-group-convention


Our team chose symbols for brevity sake for our prefixes.

Underscore _VariableName means it’s an IIS infrastructure configuration.  We deploy to many shared IIS environments with the Manage IIS and Deploy IIS tasks.

Caret ^VariableName means it comes from a variable group.

No prefix, it’s a local variable.

Use Release scope for variable groups mapped to stages

I covered the next two conventions in an earlier post.  If your variable groups are mapped to stages you can use one definition in your pipeline variables scoped to Release.  The stages will properly map.

This:

variable-group-pipelinevar-reference-to-release

Not this:

variable-group-pipelinevar-reference-per-stage

How to override a variable group value

Given a pipeline variable mapped to a variable group is using the Release scope one can then overwrite a stage by adding a staged scoped variable and value. 

override-variable-group-release-value


Doing this makes it obvious we are overriding and makes it easy to remove and go back to the variable groups value.  It also avoids having to look through logs to see what was used if one checked the “settable at release time” option. 

Don’t lock

As a starting point, don’t lock if it isn’t a secret.  If it is a secret, consider this carefully.  My reason isn’t to thwart proper security concerns.  I’ve seen too many times where the actual secret was not stored elsewhere and/or where it was stored elsewhere was no longer accurate.  Once it’s locked, you will never see it again in DevOps.  Troubleshooting this especially if one does not have a way to verify the configuration is rough.

Instead of locking use Azure Key Vault

If you determine that the secret must be protected from casual viewing then please use a mechanism where the source feeds the variable group in a secure fashion.  Azure Key Vault is a great option for this.

Create a Developer Wiki for the Variable Groups

Document the variable groups, share it, make sure the developers know about them.  I’ve ran into countless cases of pipeline variables that were setup with local values when a variable group existed.


Learn More

Add & Use variable groups
Connecting Azure Key Vault to a variable group

To the node pro this post will not surprise.

TLDR; If NODE_ENV is set to production npm install and npm ci commands won’t install developer dependencies without passing --include=dev  (-–also=dev for npm older than v7).  


Why would you install developer dependencies in a Production environment?  Agree, 100%.  However, we do, gulp in particular. Not that I’m happy about it either.


This functionality isn’t hidden but it is interesting how the instructions returned when we tried to use a developer dependency (gulp) ended up throwing the team for a loop and a few wasted cycles.


Story Time


It’s the age ole story of unwritten knowledge within the company, lack of expertise and lack of keeping things up to date.

For us, it all started about seven years back during an ITS merger. The merger brought several disparate development teams into one and those teams used different development stacks (Java, Node, .Net).  The team members from those teams continued to maintain the apps until there were no more staff from those teams left.  Some apps were retired, some were rewritten to the chosen platform (.Net), some remained entombed in times gone by.  We did a fairly good job with documentation and even had recorded trainings from the staff leaving.

Several years went by, apps happily running, a few glitches here and there with failures that we rectified through normal operational tasks. I’m a node newb and so are my fellow team members who had the task of updating one of these node applications.  The process to update the node app is not what I would consider modern.  There is no CI/CD chain.

The process to update the application on the server(s) was:

1. git pull to get the latest code.

2.  npm ci

3.   gulp to do all the needed minifying, bundling, copying, etc.

The typical local development machine steps or on a build server.  We happily updated the code needed and dutifully followed the process laid out on our development and test servers.  We moved to the production systems and when we ran the gulp step we got:


Local gulp not found in

Try running: npm install gulp


Scratching our heads, we compared dev/test and prod environments.  All seemed properly aligned.  We poured over our docs again not finding anything.  So we ran `npm install gulp` and we got: updated 1 package in 26.605s


To our surprise when we manually ran our `gulp` command we still got: Local gulp not found in application folder Try running: npm install gulp


So after more research and not finding anything we tried a force install.  No change. We could see that gulp wasn’t in the node_modules folder.


The Aha Moment


We found in the npm-ci docs in the omit parameter section the following two pieces of information.

“The default state is to omit developer dependencies when NODE_ENV is set to production.”  Makes sense except it’s telling us it updated gulp?

“Note that these dependencies are still resolved and added to the package-lock.json or npm-shrinkwrap.json file. They are just not physically installed on disk.”


Totally not what we would expect to occur. Long story short, it resolved and updated the package-lock.json file but didn’t actually install.


To install developer dependencies when in production mode pass `—include=dev` ( `-–also=dev` for npm older than v7) to have it REALLY install your developer dependencies to disk.

Learn More

https://docs.npmjs.com/cli/v9/commands/npm-ci#omit

https://worknme.wordpress.com/2021/12/19/how-to-install-dev-dependencies-in-node-env-production/

<< Older Posts