Search

Wednesday, December 14, 2016

Building a Reusable Windows Configuration - A Practical Guide

Part I - Introduction and Setup

Ever since I learned of the existence of Ghost (now owned by Symantec), I have used the concept to create a custom Windows configuration. The process was as follows:

  1. Partition the physical drive in two partitions.
  2. Install Windows on the first partition. Use the second partition for data.
  3. Install drivers.
  4. Install and configure applications.
  5. Use the system for a few days.
  6. Make an image of the Windows partition and store in a safe place.
  7. Use the machine for 3-6 months, re-image back from the image.
  8. Use the system some more, make adjustments and create an incremental image.

This model works, and I still use it. Instead of Ghost, I now use Acronis True Image. True Image is more modern, has compression, provides incremental/differential backups, and more. However, installing and configuring applications (#4 above) has always been a manual and lengthy process. In this guide, I will show you a practical way to create a reusable Windows configuration, one which you can apply on any machine.

Goals

  • Create a script that can run on any Windows machine.
  • Ability to install/remove Windows features.
  • Ability to apply Windows updates.
  • Ability to install our applications of choice either from a local installer or the web.
  • Ability to restore settings for the applications of our choosing.
  • Ability to restore various Windows profile settings.

Let’s start!

We will be using the following tools:

NuGet

NuGet is a package manager that is widely known in the Microsoft world. It’s a way to package and redistribute source code, that integrates flawlessly into Visual Studio. NuGet packages have the .nupkg extension, and contain the package manifest (a file called NuSpec using the .nuspec extension, an XML file), and usually the PowerShell script to install the source code/software. The NuGet package file is just a zip file with some metadata attached to it.

Chocolatey

Chocolatey takes the NuGet concept to the next level. Chocolatey builds on top of NuGet and allows us to use NuGet packages to install not just source code inside our development environment, but also Windows applications and features. If you don’t have Chocolatey, go to their page and install it by running the PowerShell provided on their page. The Chocolatey package gallery has over 4000 packages ready to install. All you do is:

# Install Google Chrome
choco install googlechrome

PowerShell

Powershell is a powerful scripting language built on top of the .NET framework. Think BAT or Linux bash but with the whole .NET framework behind it.

By itself Chocolatey will let you install all the applications you want, but we want more than just applications. Also, by default Chocolatey packages will only install from the web (redistribution rights and all). For our purpose, we want to download the installers once, and be able to run everything locally. To provide some of that functionality, we will need to extend Chocolatey with extensions. Chocolatey extensions are just PowerShell modules that Chocolatey loads and we can put our common functionality in those modules. A Chocolatey extension itself is a NuGet/Chocolatey package. Let’s build our extensions first.

Create a new directory, I called mine Boyan-Choco.Extensions, inside the directory create a new .nuspec file, I called mine Boyan-Choco.extension.nuspec. This file will contain the metadata for our extensions. The important metadata are the ID, and the version.

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2015/06/nuspec.xsd">
  <metadata>
    <id>Boyan-Choco.extension</id>
    <version>1.0.0</version>
    <title>Install Helpers</title>
    <authors>Boyan Kostadinov</authors>
    <projectUrl>https://github.com/Boyan-Kostadinov/Packages</projectUrl>
    <owners>Boyan Kostadinov</owners>
    <packageSourceUrl>https://github.com/Boyan-Kostadinov/BoxStarter</packageSourceUrl>
    <summary>Common Install Functionality</summary>
    <description>
    Chocolatey installation extensions
    </description>
  </metadata>
</package>

The actual PowerShell modules will reside in the extensions directory (it must be called extensions), so create one inside the same directory where the NuSpec file is. Let’s create some essential PowerShell functions. We will split functionality into separate PowerShell modules (PowerShell modules end with the .psm1 extension). Let’s create the following:

Note: I am not going to cover every function in every module. The full source can be found under my GitHub repository at https://github.com/Boyan-Kostadinov/BoxStarter/tree/master/Boyan-Choco.extension.

Let’s implement some of the functions in the ChocoHelpers module.

First is the Invoke-Commands function. It will take a file, and a line template, and for each line that doesn’t start with a # (which is the PowerShell character for a comment), it will replace the ##token## with the current file line, and execute that expression by using the PowerShell Invoke-Expression

function Invoke-Commands([string] $file, [string] $commandTemplate) {
    try {
        foreach ($line in Get-Content -Path $file | Where-Object {$_.trim() -notmatch '(^\s*$)|(^#)'})
        {
            $commmand = $commandTemplate.replace("##token##", $line)

            Write-Host "Running: $commmand"

            Invoke-Expression $commmand
        }
    }
    catch {
         Write-Host "Failed: $($_.Exception.Message)"
    }
}

This way we can pass a file with Chocolatey packages, one per line, and have Chocolatey install each one. Our file would look like so:

GoogleChrome
JRE8
MsSQLServer2014Express --packageParameter "….." --installArguments "…."

And the template we will pass to the above function will be:

choco install ##token## --execution-timeout 14400 -y

Where we call choco (shortcut for Chocolatey) with install and execution timeout of 4 hours. We also pass the -y flag so Chocolatey does not asks us for confirmation for every package.

The function that abstracts that is fairly simple. It takes only the file containing our Chocolatey packages:

function Install-Applications([string] $file)
{
    Write-Host "Installing Applications from $file"

    if ($env:packagesSource) {
        $packagesSource = "-s ""$env:packagesSource;chocolatey"""
    } 

    Invoke-Commands $file "choco install ##token## --execution-timeout 14400 -y $packagesSource"
}

Onto the next helper, the FileHelper module.

The most important function here is the Get-ConfigurationFile function. This function will be used to get our Chocolatey package configuration file. The file can be a local file, embedded in the package, or a URL of a remote file, that will be downloaded. The function will also take a parameter for a default configuration file, in case one wasn’t provided by the user. Here is the full listing:

function Get-ConfigurationFile()
{
    param(
        [string] $configuration,
        [string] $defaultConfiguration
    )

    if ([System.IO.File]::Exists($configuration))
    {
        return $configuration
    }

    if (($configuration -as [System.URI]).AbsoluteURI -ne $null)
    {
        $localConfiguration = Join-Path $env:Temp (Split-Path -leaf $defaultConfiguration)

        if (Test-Path $localConfiguration)
        {
            Remove-Item $localConfiguration
        }

        Get-ChocolateyWebFile 'ConfigurationFile' $localConfiguration $configuration | Out-Null

        return $localConfiguration
    }

    return $defaultConfiguration
}

The FileHelper also has unzip functions, path helpers, etc., nothing interesting to write home about.

The most interesting, and most useful of the helpers is the InstallHelper. I will cover the Get-InstallerPath, Install-LocalOrRemote and the Install-WithScheduledTask functions.

The purpose of Get-InstallerPath is to look for a setup executable, installer executable, installer path, an ISO image or a URL provided in the arguments. It uses the Get-Parameters function to parse the Chocolatey provided parameters, and then looks for one of the following in this exact order:

1. Setup path - An executable path provided with the parameter **/setup="Path.To.Exe"**
2. Installer path - An executable that unpacks the setup for the package.
3. A package installer - A combination of the path defined in **$env:packagesInstallers** and the **file** argument.
4. An ISO path - The path to the ISO image provided with the parameter **/iso="Path.To.Exe"**
5. A URL to the installer - Provided in the url argument. The file is downloaded and stored in the file argument.

The point of all this is to account for all possible scenarios when it comes to installing a package. This will make more sense when we get to the package creation stage.

Once we figured out where the executable of the package is, Install-LocalOrRemote, is really simple. It gets the file from the file argument and it uses the Chocolatey function Install-ChocolateyInstallPackage to install the package. The full listing:

function Install-LocalOrRemote()
{
    param(
        [Hashtable] $arguments
    )

    $arguments['file'] = Get-InstallerPath $arguments

    if ([System.IO.File]::Exists($arguments['file']))
    {
        Write-Debug "Installing from: $($arguments['file'])"

        Install-ChocolateyInstallPackage @packageArgs

        CleanUp
    }
    else {
        throw 'No Installer or Url Provided. Aborting...'
    }
}

Things get slightly more interesting with the Install-WithScheduledTask. Because some installers (I’m looking at you Spotify, shame on you!) don’t allow you to install the application if you are running as administrator, we have to come up with clever hacks to get around that. This function uses the Windows Task Scheduler to schedule a task with the installer executable, start that task and then delete the scheduled task. The StartAsScheduledTask function is in the SystemHelpers module. Here is the full listing:

function StartAsScheduledTask() {
    param(
        [string] $name,
        [string] $executable,
        [string] $arguments
    )

    $action = New-ScheduledTaskAction -Execute $executable -Argument $arguments
    $trigger = New-ScheduledTaskTrigger -Once -At (Get-Date)

    Register-ScheduledTask -TaskName $name -Action $action -Trigger $trigger
    Start-ScheduledTask -TaskName $name
    Start-Sleep -s 1
    Unregister-ScheduledTask -TaskName $name -Confirm:$false
}

Next comes the RegistryHelper module. It contains the following registry related functions:

  • Test-RegistryValue - Checks if the registry value exists at the provided registry path.
  • Import-RegistrySettings - Given a file system path, it iterates over all the .reg files found and imports them with the Windows registry utility regedit.
  • Import-RegistryFile - Given a file, executable and a process name, it starts the executable, kills the process with the process name provided, and then imports the registry file provided in the file argument. The purpose is to start the program, and have it create it’s default settings, kill it, and then import the provided registry settings.

The SystemHelpers module is nothing special. It defines the StartAsScheduledTask function and the IsSystem32Bit function.

The WindowsFeatures module is slightly more interesting. It defines a way to install Windows features by using Chocolatey (and to be exact the special source in Chocolatey called WindowsFeatures).
- Enable-WindowsFeatures - Provided a file path, passes that file to Invoke-Commands, using the template choco install ##token## -r -source WindowsFeatures -y.
- Disable-WindwosFeature - Does the opposite of Enable-WindowsFeatures by using choco uninstall.
- Enable-WindowsFeature - Enables a single Windows feature where the argument is the feature name. Uses Get-WindowsOptionalFeature and Enable-WindowsOptionalFeature, which are only available in Windows 2016 and Windows 10. To get the name of the features you can run choco list -source WindowsFeatures.

Now that we have all the modules defined, we can build our extensions package by navigating to the directory and running

choco pack Boyan-Choco.extensions.nuspec

That will produce a Boyan-Choco.extension.1.0.0.nupkg, which you can install by hand with

choco install Boyan-Choco.extension -source P:\ath\To\NuPkg\Directory

I say by hand, because we will not be installing the extension package, but instead it will be a dependencies that gets installed as part of the packages we will create.

Continue to Part II - Creating Your Packages…Coming Soon

Wednesday, August 31, 2016

To Var or not to Var

 

I got asked today about my use of “var” in .NET. As it happens in every debate, I forgot all reasons of why I use “var.” After thinking about it, the main reason is, and always will be, “I like var.;” it’s concise, it’s easy to read, and it saves me time typing more complex types. I have never found myself wondering what type the variables is, so I don’t care about the way I declare its type. I only care about how I use that type. But unlike Jim Jeffries common argument for guns: “Fuck you! I like guns!” (as described in his standup on YouTube: https://www.youtube.com/watch?v=0rR9IaXH1M0), which I agree is a perfectly valid argument for doing something, there is more reasons to using “var.” So I will talk about each one in turn.

  1. It’s as readable as the implicit type declaration. It makes no difference whether you declare your type before or after the variable. The following are the same.

    List<Widgets> widgets;

    var widgets = List<Widgets>();
    

  2. I don’t have to repeat myself. I am lazy, even more so when I write code. I can “get two birds stoned at once!” So doing this:

    var widgets = List<Widgets>();

    Is easier to write than this:

    List<Widgets> widgets = new List<Widget>();

  3. I get the benefit of my variable never being null! No more “Object reference not set to instance of an object!” Sure, this is not a problem for simple types like string, int, but it’s a pain in the ass for and lists like List<Widget>. How many times have you declared your list, forgot to initialize it, then try to put junk in it?! I know I have!
  4. Using “var” is only allowed in local scope, which means that every time I declare something with “var”, you know it will be only available in the local scope. Thank you Captain Obvious!
  5. I don’t care what the compiler does with it when it compiles! It’s none of my business.

A Sidebar on Code Readability

The bigger issue with readability that most folks miss, is how you name your variables and how you structure your code. Here are a couple of things that make code much harder to read:

  • Variable names after single letters. “L” or “R” might mean something to you if you wrote the code, but to anyone new looking at it, it might as well be Greek. Which is great if you are Greek but not so much for everyone else. How about “Log” or “Result?”
  • Function size. If your function is bigger than a page, and I have to scroll to read it, you are doing it wrong.
  • Inconsistent capitalization. A widely accepted standard is that parameters are camelCase, private _variables are small case and can start with “_” and public Properties are title case.
  • Too many parameters. If a function has more than 2-3 parameters, it’s time to make a object to pass those parameters. Readability goes out the window when function parameters start spanning multiple lines.
  • Silly patterns that somehow still persists. By that I am talking about variables having the type letter in the name, such as “bFlag” or “iNumber.” Is “b” byte, is it bool? A better choice would be “IsFlag.” There is nothing to be gained by specifying the type in the variable name. The IDE already knows the type, and you have all the intellisense associated with that type.
  • Multiline strings. Inline SQL is the poster child for this. A slightly better approach is put your multiline string at the bottom of your class and use some string replacement to fill out it’s values at runtime. Use verbatim strings if you want multiline strings that you can copy/paste in other places. Or better, use an ORM, that’s what they are for!

Credits

Credit where it’s due, in no particular order: Burton Rheutan, Endurance Idehen, James Allen.

Wednesday, March 23, 2016

Installing Software on Windows the Easy Way (Using Chocolatey)

I've been using Chocolatey for a while now, and today I want to share the coolness, for those of you that haven't heard of it or used it. It’s much like apt-get for Windows (automatic way to install software from the web).

You install it by running this on the command line:

@powershell -NoProfile -ExecutionPolicy Bypass
-Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))"
&& SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

After that, let’s say you want to install Chrome, it’s simple as (running from CMD)
choco install chrome

or EverNote:
choco install evernote

or Node:
choco install nodejs.install

You just have to remember to run CMD as administrator. Bypass the license prompt by using the "-y" command argument. You can install any of the 3779 packages it supports. They are listed at https://chocolatey.org/packages

Monday, March 14, 2016

Using NUnit 3.2 with ReSharper Ultimate 10


I just upgraded my ReSharper to 10 in the hope that it will fully support NUnit 3.2 (the most current version as of the time of me writing this). However that didn’t turn out to be the case. I still had to Google around to find why ReSharper won’t run my tests while using NUnit 3.2. The solution turned out to be simple – install the ReSharper NUnit runner for NUnit 3 through the package console and restart Visual Studio!

Install-Package NUnit.ReSharperRunner3

Sunday, March 13, 2016

Poor Man’s Batch Encoding with HandBrake

I wanted to have a way to batch encode a directory (or even multiple directories) full of files without having to use the HandBrake interface to queue each directory. Ideally, I will do this in .NET in the long run but for now, here is a simple way to walk through all the directories and encode all files with the same HandBrake settings.

How it Works

  1. Deletes any existing files with the names “_encode_info.log” and “_encode_progress.log”
  2. Gets all the files in the directory and sub directories where it runs, ignoring itself
  3. For each file it finds:
    1. it gets the name of the directory the file is in, creates a directory with the same name under the “destinationPathRoot.”
    2. Sets the name of the output file to be under the newly created directory under “destinationPathRoot”
    3. If the file already exists, it sets it’s name to have the current date and time (to avoid overwriting)
    4. Calls the EncodeVideo function with the source file path and destination file path
    5. If the destination file path exists after the encoding, it deletes the source file path


Setup

  • handBrakeExecutable – Set the path to the HandBrakeCLI executable
  • destinationPathRoot – The root of the path where to put the encoded files


How to Use It

Copy the contents below and paste it in a .bat file. I name mine according to the HandBrake arguments I’m using. For example “_encode-Q25-1Audio,bat” (meaning using variable quality of 25 with 1 audio stream). Copy the .bat file in the root of the directory where your videos reside. Run it.

Code

@echo off
cls
setlocal enabledelayedexpansion enableextensions

rem Set the path to the Handbrake executable
set handBrakeExecutable=Path\To\HandBrakeCLI.exe

rem Set the root path where files will be encoded to
set destinationPathRoot=Path\To\Done\Videos

if exist "%CD%\_encode_info.log" del "%CD%\_encode_info.log"
if exist "%CD%\_encode_progress.log" del "%CD%\_encode_progress.log"

rem Get all the files in the directory and sub-directories
for /f "tokens=* delims=|" %%G in ('dir /s /a-d /b *.*') do if not %%~xG==.bat (
    rem Set the source file path
    set sourceFilePath=%%G

    rem Get the current directory and remove the trialing slash
    set currentDirectory=%%~dpG
    set currentDirectoryNoTrailingSlash=!currentDirectory:~0,-1!

    rem Get only the name of the current directory
    for %%f in (!currentDirectoryNoTrailingSlash!) do set currentDirectoryName=%%~nxf

    rem Create the path to the destantion path based on the current directory name
    set destinationPath=!destinationPathRoot!\!currentDirectoryName!

    rem if the destination doesn't exist, create it
    if not exist !destinationPath! (
         mkdir !destinationPath!
    )

    rem Process the file
    echo File: %%G

    rem Set the destination file based on the destination path and the current file name
    set destinationFilePath=!destinationPath!\%%~nG.mp4

    rem If the filename already exists, append the current date and time to it
    if exist destinationFilePath (
        rem Get the current date and time in MMDDYY_HH_MM_SS format
        for /f "tokens=1-8 delims=:./ " %%G in ("%date%_%time%") do (
            set currentDateTime=%%G%%H%%I_%%J_%%K
        )

        set destinationFilePath=!destinationPath!\%%~nG-!currentDateTime!.mp4
    )

    rem Call Handbrake to encode the file
    call :EncodeFile "!sourceFilePath!", "!destinationFilePath!"

    rem Delete the source file if the destination exists
    if exist !destinationFilePath! (
        del "!sourceFilePath!"
    )
)

:EncodeFile
setlocal

rem echo Source: %~1
rem echo Destination: %~2

rem Call HandBrakeCLI to encode the file
"!handBrakeExecutable!" ^
-I "%~1" ^
-o "%~2" ^
-f mp4 ^
-O ^
--decomb="fast" ^
--crop 0:0:0:0 ^
--strict-anamorphic ^
--modulus 2 ^
-e x264 ^
-q 25.0 ^
--cfr ^
-a 1 ^
--encoder-level="4.0" ^
--encoder-profile=high ^
1> _encode_progress.log ^
2> _encode_info.log

endlocal
// //]]>