User Secrets – What Are They And Why Do I Need Them?

Today’s header image was created by Kristina Flower at Unsplash

Just a quick caveat before we begin.

This article was written before .NET Core 2.0 was RTM’d (it was still in Preview 2 when I wrote this article). Almost everything is the same, but some of the screenshots and code snippets make it obvious that I’d written the article with a 1.x code base.

Again, the process is the same. Just remember to include version 2.0 of theMicrosoft.Extensions.Configuration.UserSecrets package.

---

We’ve all been there: added some functionality which talks to an external API which requires a secret key, hit commit, then days later wondered why out secret key is being used to authorise applications we didn’t write.

In fact, more developers than we care to think about have done this. How do I know? Take a look at the following searches on GitHub:

• search key: “Remove Password“
• search key: “Remove apikey“
• search key: “Remove connection string“

in fact, Github have a page devoted to showing developers how to remove sensitive (passwords, api keys, etc.) from their commit history.

which you can read here

It’s a big issue, as you can see. It’s all too easy to accidentally commit sensitive information to a repository. Which is where User Secrets comes in.

If you work on open source stuff (even if you work on closed source stuff), you don’t want to be storing sensitive information in configuration files. Especially if they are stored on disk in plain text.

I’m thinking ini, xml and json config files here. But anything plain text is not safe

If some nefarious person gains access to the server that the source code is stored on, then they have access to all of your sensitive information. Granted, if someone gets access to the server then you have slightly bigger issues than just your application’s sensitive information.

Unless it’s open source, in which case anyone can search for said sensitive information right there in your code base.

Just like the searches I listed above

Hopefully the folks who have had to issue those commits have changed the passwords and api keys that they were using, but that doesn’t always happen.

User Secrets

Anyway, what does user secrets have to do with source control?

User Secrets are specific pieces of sensitive information which need to be kept out of source control: api keys, connection strings, administrator passwords, etc.

although what you’re doing storing passwords in source code, is beyond me

User secrets aren’t new to .NET Core, it was first committed to the ASPNET repository in March 25th 2015 and moved to the Configuration repository in September of 2016.

the glory of open source, eh?

The whole point of User Secrets is that developers can store their sensitive data for the application they’re working on (passwords, api keys, connection strings, etc.) in a file separate from the code tree. This means that they cannot be accidentally committed into source control.

How Does It Work?

Developers use a command line tool (or a tool within Visual Studio) called the Secret Manager to add secrets for a specific application to a directory which is semantically different (and far away) from where the source code is being written.

Secrets are then added to a sub-directory within the developer’s home directory.

I just read that back, and I’m sorry. It’s a difficult sentence. Let me try again

A hidden directory is added to the developer’s home directory called .microsoft and a sub-directory is added to that called usersecrets. Within this directory, sub-directories for all application projects with user secrets are created.

this is done by the secret manager tool; we’ll see how in a moment.

Within each application secrets directory, a file called secrets.json is stored. Any secrets added to a project via the secret manager tool will be added to this file.

The contents of the secrets.json file is plain text, in fact Microsoft has this to say about the contents of the file and their security:

The Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store. It is for development purposes only. The keys and values are stored in a JSON configuration file in the user profile directory.

Source: https://docs.microsoft.com/en-us/aspnet/core/security/app-secrets#secret-manager

Where You Would Use Secrets

Local development only.

Seriously, local development only. Never in staging or production. Just local development. Again, here are Microsoft’s words on User Secrets in production:

The Secret Manager tool is used only in development. You can safeguard Azure test and production secrets with the Microsoft Azure Key Vault configuration provider.

Source: https://docs.microsoft.com/en-us/aspnet/core/security/app-secrets

For staging and production, you can use the Microsoft Azure Key Vault or appsettings.Staging.json and appsettings.Production.json files. These can be generated or supplied to your build system, and pushed to the server.

securing the server is out of the scope of this post, and entirely dependant on the Operating System on the server

Creating and Using User Secrets

As with almost everything else I’ve written about, we’re going to build an example application to show off the usage of User Secrets.

everything’s easier to learn with an example, right?

We’ll leverage the built in MVC template, add a user secret to it and consume it while running in development. We’ll then run the application in staging and production to see how the same code can be used to load our sensitive configuration data while in those environments.

a lot of the hard work has been abstracted away for us by the .NET Core team, as we’ll see

I do everything with VSCode and the terminal, but I’ll include a section for adding user secrets from within Visual Studio 2017, too.

if you want to skip all of this and jump straight to the code, you can clone it from this GitHub repo

The application project

Head over to the terminal and issue the following command:

This will create an MVC project in a directory called userSecrets. Open that directory with VSCode and you’ll be presented with something like the following:

The first thing we need to do is add a user secrets Id to the csproj file. So we need to edit the csproj to include the following in the PropertyGroup section:

I’ve given a pseudo-random Id and appended it to the name of the application, but you can use whatever convention you’d like

The next thing we need to do is include the UserSecrets package, this is found at Microsoft.Extensions.Configuration.UserSecrets:

psst, for .NET Core 2.0 users: use version 2.0.0

And finally, to access the command line tooling, we need to add a reference to that too. The command line tools are available in the Microsoft.Extensions.SecretManager.Tools package:

psst, for .NET Core 2.0 users: use version 2.0.0

As a side note: I tend to add the command line tools to a separate section in my csproj, but feel free to add them where ever you feel works best for you.

Adding Secrets

Now that we’ve added the correct tooling, we need to restore the packages that we want to consume:

In order to check that the user secrets tooling has been installed correctly, we can issue the help command and see how to use it:

If we get the following (or similar), then we haven’t installed restored the packages correctly:

However, output along the same lines as this shows that the tooling has been installed correctly (and shows how we can use it)

We’ll add a user secret

in the documentation, this is referred to as setting a value

called SuperStrongPassword by issuing the following command:

Except that wont work (at least on Unix-like machines), and will give the following error

This is because we have to escape the bang character in order to save it.

exclamation marks in the Unix world are referred to as bangs

This is purely a Unix thing, so you shouldn’t be affected if you’re running Windows. Either way, here is the fixed command for Unix-like operating systems:

If we need to check the keys and values of all of our User Secrets, we simply issue the following command:

Which in this instance should come back with:

or whatever you set the value of SuperStrongPassword to

What About Visual Studio?

In Visual Studio, there is a User Secrets Manager which is accessed by right clicking on the project (which represents the csproj) that you want to add secrets for, and selecting Manage User Secrets.

This will create the secrets.json file and open it in Visual Studio:

Then it’s a case of typing your secrets manually and saving the file.

Location of User Secrets

I touched on this earlier, but I feel like covering it again.

The User Secrets file (secrets.json) for our project will be found at one of the following paths (dependant on the operating system that you are running):

• Windows: %APPDATA%\microsoft\UserSecrets\<userSecretsId>\secrets.json
• Unix: ~/.microsoft/usersecrets/<userSecretsId>/secrets.json

So in our example, the user secrets will be found at one of these two paths:

• Windows: %APPDATA%\microsoft\UserSecrets\userSecrets-c23d27a4-eb88\secrets.json
• Unix: ~/.microsoft/usersecrets/userSecrets-c23d27a4-eb88/secrets.json

Opening our secrets.json file, we’ll find a very simple structure:

Consuming the User Secrets

Just to re-iterate before we continue:

you should only use User Secrets in your development environment. Never in Staging or Production

In order to load our User Secrets, we need to make a few changes to the Startup.cs file. We need to:

• Add the UserSecrets namespace
• Add User Secrets to the application builder
• Read the value from our User Secrets file

Let’s take the startup.cs file and replace it with the following:

Taking a look at the relevant lines of code:

Here we’ve added the relevant namespaces.

Here we’ve added a string to hold our user secret and we’ve instructed the application builder to add the user secrets configuration to our application (we’ll consume it in a moment).

Here we’re telling our application to go get the value of SuperSecretPassword from the configuration dictionary (our user secrets have already been loaded into memory by this point) and store it in our string from earlier.

I feel like part of that paragraph is a little teaching gandma to suck eggs

Here is where we’re consuming the value of our UserSecrets. We’re creating a response which contains that value of our loaded UserSecret.

Fire the application up with the terminal

it’s important that you do it with the terminal first, as there’s a caveat here that we need to cover

Then head over to whatever port on localhost

you’ll see, in a moment, that I was given port 5000

and you should get a response from the server:

Looking back at our Configure method, we were using the value from the User Secrets or the string “Secret is Not Found” if it couldn’t be found:

Which proves that the user secret couldn’t be found. But why?

Caveat

Remember how I (and Microsoft) said that we should only use User Secrets in development? I wasn’t kidding, and we actually set our application up specifically to use them only in development:

And if you look at the output in the terminal from when we started the application:

wait, what?

Ah. Since we didn’t explicitly state which environment we’re in, .NET Core started in Production.

and this is precisely why I made you start the application from the command line

And this make sense, if you think about it. We shouldn’t have to supply a separate command line argument to start the application in production, otherwise starting the application on the server would be more complex.

To temporarily start our application in development, we need to run the following command:

This should give us the following output in the terminal:

Which means that if we refresh our browser, we should get the User Secret in our response:

As a side note, you can make the change more permanent if you want by using one of the two following commands (depending on the operating system that you’re using):

• Windows: set ASPNETCORE_ENVIRONMENT=Development
• Unix: export ASPNETCORE_ENVIRONMENT=Development

This is a permanent, system-wide, change, though. So don’t be surprised if your other applications start behaving differently.

Consuming User Secrets in a Controller

All of that is all well and good, but how do we consume the contents of our User Secrets in a Controller (or some other) method?

It turns out that it’s quite easy to do.

thank you .NET Core team for making this super easy, by the way

What we need to do is:

• Create a POCO to store the User Secrets in
• Add the UserSecrets namespace in the Startup.cs
• Add User Secrets to the application builder
• Read the value from our User Secrets file to an instance of the POCO in Startup.cs

The POCO

Since our example User Secrets are pretty simple we only need a simple model.

Let’s take another look at the secrets.json file:

So our POCO just needs to match that.

Create a Models directory in the root of the project, add a file called Secrets.cs, and add the following contents to it:

If we add more things to the User Secrets, we’ll need to add them to this POCO.

which we’ll need to remember

Adding Stuff to the Startup Class

Replace the contents of your startup.cs with the following:

We’ve covered most of this already, but I’ve highlighted the new stuff. The most important change is:

Here we’re telling the IServiceCollection to take the contents of the UserSecrets configuration and deserialise it to an instance of the Secrets class we just reated.

There’s some proper ace magic here, in that .NET Core will only deserialise keys which are in both the User Secrets file and the POCO that we’re deserialising to. If, for instance we added a key to the User Secrets which wasn’t present in the POCO, then .NET Core wont raise an error, it will just ignore it.

and the same for the opposite direction

Consuming User Secrets in a Controller

Since we’ve added an instance of the Secrets class to the IServicesCollection in the startup class, we don’t need to do very much to consume it in a controller (or anywhere else within the application). We’ll leverage .NET Core’s built in dependency injection to ensure that it gets added to our controller, then consume it.

Let’s use the HomeController’s About method as our example, since it already exists and it’ll be faster to wire it up.

not that it takes a long time to wire it up at all

Replace the contents of your HomeController with the following:

I’ve highlighted the important lines, so let’s take a look at them:

Here we’re using .NET Core’s built in dependency injection to make sure that we get our Secrets class, which was populated in the startup class.

Here we’re consuming our Secrets class.

That’s all we need to do, so let’s spin up out application and give it a whirl:

we’ll explicitly test in development first

Then head to Home/About and you should get something along the lines of the following:

What about in Production? Let’s try that now:

Refreshing the About page should give us something like:

We actually preempted this in the About method. Let’s take a look:

it’s almost as if I knew

What About Production?

Cast your mind back to when I said:

For staging and production, you can use the Microsoft Azure Key Vault or appsettings.Staging.json and appsettings.Production.json files

quoting oneself is quite pretentious, you know

That’s what we’ll do, we’ll add an appsettings.Production.json file to the root of the project and use that. We’ve already got the code which will read it for us, which is found in the startup class:

We’re already reading an appsettings.Production.json file (if present) and attempting to deserialise a relevant part of it to an instance of the Secrets class.

seriously, .NET Core does so much heavy lifting for us

Create a file called appsettings.Production.json file in the root of the project and paste the following into it:

The contents of this file should match the appsettings.Development.json file, but with one important difference:

Restart the application (in production):

And head back to Home/About and this should happen:

And Staging?

Do exactly the same as above, but create an appsettings.Staging.json file.

and chose a different value for SuperStrongPassword, obviously

Again, you’ll want to either have these files already on your production server, or generated by the build action. You are using a build server, right? It’ super easy to set something like AppVeyor up specifically for this.

in fact, I’ve already written an article about using AppVeyor for building and deploying .NET Core applications

---

Conclusion

We’ve seen the problem that User Secrets were designed to solve, how to create them, and how to consume them in your application. We’ve also seen how appsettings should be used in place of UserSecrets when outside of development (i.e. in Staging and Production).

We’ve also seen that .NET Core does all of the hard work of figuring out which files to deserialise, in which situations.

Have you used User Secrets in any of your .NET Core applications yet? If so, what did you think of them? Did they seem as amazing and magical to you as they did to me? If you haven’t used them would you be more willing to use them now that you’ve seen how to do it? Let me know in the comments and let’s keep the conversation going.