Configure WinINET proxy server with PowerShell

Configure WinINET proxy server with PowerShell

A proxy server is used to allow applications and web browsers to communicate with the internet. There are multiple benefits. One of them is caching of the results. This is less important then it used to be. You also control what sites can be visited and that you have a log of where the traffic is going to. You can use a transparent proxy which means that you do not need to configure your hosts with proxy settings. The firewall sends all the internet bound traffic (HTTP/HTTPS) to the proxy server. With a standard proxy you need to tell the hosts and applications where to go and, optionally for what sites to bypass the proxy server. There are different options and libraries to configure the Windows proxy settings. Here we focus on how to Configure WinINET proxy server with PowerShell.

Why? Primarily because I wanted to automate this. Secondly, I needed a solution that was easy and fast in a non-domain joined / work group scenario.

How to configure proxy server settings

There are basically 3 ways to define proxy settings on a Windows host.I

  • Applications using the WinINET library.  WinINET, an API, is part of Internet Explorer and can also be used by other applications. Applications leveraging the WinINET API take over the proxy settings configured in Internet Explorer. You set these per user or per machine. In the latter case only a user with administrative rights can set or change the proxy server settings. We’ll look at PowerShell to configure this for us.
  • Applications using the WinHTTP library. WinHTTP is the best choice for non-interactive usage. Prime examples are windows services or custom applications. WinHTTP does not use the proxy settings from WinINET unless you import them. This is the one we need to configure for applications that do not use WinINET. If you do not do so you will run into issues when blocking direct internet access. Settings in WinINET are ignored. Which means the proxy is not by an application or service using WinHTTP. You set this via netsh winhttp set proxy <proxy>:<port>. You can reset this via netsh winhttp reset proxy. If you want to import the WinINET setting use netsh winhttp import proxy source=ie
  • Applications that have their own proxy settings. In this case you configure the settings in the application itself. Some applications like Firefox use the systems settings by default but you can also define Firefox specific proxy settings.

Where to configure the proxy settings

Bar the options to automatically detect the proxy setting or using a script (GPO or registry editing) you can manually configure the settings for WinINET.

I have stopped using Internet explorer and as such I avoid using it to configure the WinINET settings. For that I prefer to use the Windows, Settings, Network & Internet, Proxy. The result is the same and on modern OS versions I disable Internet Explorer.

Configure WinINET proxy server with PowerShell
Manual Proxy configuration

If you want to set them machine wide you can do so via a GPO or registry editing.

Via GPO: Computer Configuration\Administrative Templates\Windows Components\Internet Explorer\Make proxy settings per-machine (rather than per user)
Via Registry:HKLM\Software\Policies\Microsoft\Windows\CurrentVersion\Internet Settings
DWORD: ProxySettingsPerUser = 0

But bar using GPOs setting Proxy setting manually can be tedious, especially when you want to clean out old settings and have multiple profiles on the hosts. So I threw together a PowerShell solution to use in work group environments. This took some research and testing to get right.

$ProxyServer = "192.168.2.5:3128"
$ProxyBypassList = "192.168.2.3;192.168.2.4;192.168.2.5;192.168.2.72;<local>"
$TurnProxyOnOff = "On"
$ProxyPerMachine = $False

<#
$ProxyServer = ""
$ProxyBypassList = ""
$TurnProxyOnOff = "Off"
$ProxyPerMachine = $False
#/#>

#Example: Set-InternetProxy "mproxy:3128" "*.mysite.com;<local>"
function Set-InternetProxy($ProxyPerMachine, $TurnProxyOnOff, $proxy, $bypassUrls) {
    if ($TurnProxyOnOff -eq "Off") { $ProxyEnabled = '01'; $ProxyEnable = 0 } Else { $ProxyEnabled = '11'; $ProxyEnable = 1 }

    $regPath = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings"
    $proxyBytes = [system.Text.Encoding]::ASCII.GetBytes($proxy)
    $bypassBytes = [system.Text.Encoding]::ASCII.GetBytes($bypassUrls)
    $defaultConnectionSettings = [byte[]]@(@(70, 0, 0, 0, 0, 0, 0, 0, $ProxyEnabled, 0, 0, 0, $proxyBytes.Length, 0, 0, 0) + $proxyBytes + @($bypassBytes.Length, 0, 0, 0) + $bypassBytes + @(1..36 | % { 0 }))
                       
    if ($ProxyPerMachine -eq $True) { #ProxySettingsPerMachine         

        New-ItemProperty -Path $regPath -Name 'ProxySettingsPerUser' -Value 0 -PropertyType DWORD -Force #-ErrorAction SilentlyContinue

        #Set the proxy settings per Machine
        SetProxySettingsPerMachine $Proxy $ProxyEnable $defaultConnectionSettings
                       
        #As we are using the per machine proxy settings clear the user settings, tidy up.
        #This is done for all profiles found on the host as well as the default profile.
        ClearProxySettingPerUser

    }
    Elseif ($ProxyPerMachine -eq $False) { #ProxySettingsPerUser
        New-ItemProperty -Path $regPath -Name 'ProxySettingsPerUser' -Value 1 -PropertyType DWORD -Force #-ErrorAction SilentlyContinue
        #write-Host "we  get here"

        #Set the proxy settings per user (this is done for all profiles found on the host as well as the default profile)
        SetProxySettingsPerUser $Proxy $ProxyEnable $defaultConnectionSettings
            
        #As we are using the per user proxy settings clear the machine settings, tidy up.
        ClearProxySettingsPerMachine

    }          
}

function SetProxySettingsPerUser($Proxy, $ProxyEnable, $defaultConnectionSettings) {
    
    # Get each user profile SID and Path to the profile
    $UserProfiles = Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\*" | Where { $_.PSChildName -match "S-1-5-21-(\d+-?){4}$" } | Select-Object @{Name = "SID"; Expression = { $_.PSChildName } }, @{Name = "UserHive"; Expression = { "$($_.ProfileImagePath)\NTuser.dat" } }

    # We also grab the default user profile just in case the proxy settings have been changed in there, but they should not have been
    $DefaultProfile = "" | Select-Object SID, UserHive
    $DefaultProfile.SID = ".DEFAULT"
    $DefaultProfile.Userhive = "C:\Users\Public\NTuser.dat"

    $UserProfiles += $DefaultProfile

    # Loop through each profile we found on the host
    Foreach ($UserProfile in $UserProfiles) {
        # Load ntuser.dat if it's not already loaded
        If (($ProfileAlreadyLoaded = Test-Path Registry::HKEY_USERS\$($UserProfile.SID)) -eq $false) {
            Start-Process -FilePath "CMD.EXE" -ArgumentList "/C REG.EXE LOAD HKU\$($UserProfile.SID) $($UserProfile.UserHive)" -Wait -WindowStyle Hidden
            Write-Host -ForegroundColor Cyan "Loading hive" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        }
        Else {
            Write-Host -ForegroundColor Cyan "Hive already loaded" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        }                    
                    
                    
        $registryPath = "Registry::HKEY_USERS\$($UserProfile.SID)\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
        #$registryPath = "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
        Set-ItemProperty -Path $registryPath -Name ProxyServer -Value $proxy
        Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value $ProxyEnable
        Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings   

        
        # Unload NTuser.dat if it wasen't loaded to begin with.  
        If ($ProfileAlreadyLoaded -eq $false) {
            [gc]::Collect() #Ckean up any open handles to the registry to avoid getting an "Access Denied" error.
            Start-Sleep -Seconds 5 #Give it some time
            #Unoad the user profile, but only if we loaded it our selves manually.
            Start-Process -FilePath "CMD.EXE" -ArgumentList "/C REG.EXE UNLOAD HKU\$($UserProfile.SID)" -Wait -WindowStyle Hidden | Out-Null
            Write-Host -ForegroundColor Cyan "Unloading hive" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        } 
    }
        
}    

    
function SetProxySettingsPerMachine ($Proxy, $ProxyEnable, $defaultConnectionSettings) {

    #Set the proxy settings per machine (this is done for both X64 and X86)
    $registryPath = "HKLM:\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
    Set-ItemProperty -Path $registryPath -Name ProxyServer -Value $proxy
    Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value $ProxyEnable
    Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings
            
    $registryPath = "HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Internet Settings"
    Set-ItemProperty -Path $registryPath -Name ProxyServer -Value $proxy
    Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value $ProxyEnable
    Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings
}

Function ClearProxySettingPerUser () {
    # Get each user profile SID and Path to the profile
    $UserProfiles = Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\*" | Where { $_.PSChildName -match "S-1-5-21-(\d+-?){4}$" } | Select-Object @{Name = "SID"; Expression = { $_.PSChildName } }, @{Name = "UserHive"; Expression = { "$($_.ProfileImagePath)\NTuser.dat" } }

    # We also grab the default user profile just in case the proxy settings have been changed in there, but they should not have been
    $DefaultProfile = "" | Select-Object SID, UserHive
    $DefaultProfile.SID = ".DEFAULT"
    $DefaultProfile.Userhive = "C:\Users\Public\NTuser.dat"

    $UserProfiles += $DefaultProfile

    # Loop through each profile we found on the host
    Foreach ($UserProfile in $UserProfiles) {
        # Load ntuser.dat if it's not already loaded
        If (($ProfileAlreadyLoaded = Test-Path Registry::HKEY_USERS\$($UserProfile.SID)) -eq $false) {
            Start-Process -FilePath "CMD.EXE" -ArgumentList "/C REG.EXE LOAD HKU\$($UserProfile.SID) $($UserProfile.UserHive)" -Wait -WindowStyle Hidden
            Write-Host -ForegroundColor Cyan "Loading hive" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        }
        Else {
            Write-Host -ForegroundColor Cyan "Hive already loaded" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        }

        #As you are using per machine setttings erase any proxy setting for the current user.
        $proxyBytes = [system.Text.Encoding]::ASCII.GetBytes('')
        $bypassBytes = [system.Text.Encoding]::ASCII.GetBytes('')
        $defaultConnectionSettings = [byte[]]@(@(70, 0, 0, 0, 0, 0, 0, 0, 01, 0, 0, 0, $proxyBytes.Length, 0, 0, 0) + $proxyBytes + @($bypassBytes.Length, 0, 0, 0) + $bypassBytes + @(1..36 | % { 0 }))
           
        $registryPath = "Registry::HKEY_USERS\$($UserProfile.SID)\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
        Set-ItemProperty -Path $registryPath -Name ProxyServer -Value ''
        Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value 0
        Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings   
        
        # Unload NTuser.dat if it wasen't loaded to begin with.  
        If ($ProfileAlreadyLoaded -eq $false) {
            [gc]::Collect() #Clean up any open handles to the registry to avoid getting an "Access Denied" error.
            Start-Sleep -Seconds 2 #Give it some time
            #Unoad the user profile, but only if we loaded it our selves manually.
            Start-Process -FilePath "CMD.EXE" -ArgumentList "/C REG.EXE UNLOAD HKU\$($UserProfile.SID)" -Wait -WindowStyle Hidden | Out-Null
            Write-Host -ForegroundColor Cyan "Unloading hive" $UserProfile.UserHive "for user profile SID:" $UserProfile.SID
        } 
    }
}
    
Function ClearProxySettingsPerMachine () {
    #As you are using per user setttings erase any proxy setting per machine
    $proxyBytes = [system.Text.Encoding]::ASCII.GetBytes('')
    $bypassBytes = [system.Text.Encoding]::ASCII.GetBytes('')
    $defaultConnectionSettings = [byte[]]@(@(70, 0, 0, 0, 0, 0, 0, 0, 01, 0, 0, 0, $proxyBytes.Length, 0, 0, 0) + $proxyBytes + @($bypassBytes.Length, 0, 0, 0) + $bypassBytes + @(1..36 | % { 0 }))
            
    $registryPath = "HKLM:\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
    Set-ItemProperty -Path $registryPath -Name ProxyServer -Value ''
    Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value 0
    Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings
            
    $registryPath = "HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Internet Settings"
    Set-ItemProperty -Path $registryPath -Name ProxyServer -Value ''
    Set-ItemProperty -Path $registryPath -Name ProxyEnable -Value 0
    Set-ItemProperty -Path "$registryPath\Connections" -Name DefaultConnectionSettings -Value $defaultConnectionSettings
}

Set-InternetProxy $ProxyPerMachine $TurnProxyOnOff $ProxyServer $ProxyBypassList

Test the above script to Configure WinINET proxy server with PowerShell.in a VM to verify its behaviour. I hope this help somebody and probably my future self as well.

TechNine March Updates for IT Pro’s

TechNine March Updates for IT Pro’s

TechNine (https://www.technine.be/) is organizing its first user group event of 2020 on Tuesday 3 March. The TechNine March Updates for IT Pro’s. This is just a quick blog post to invite all who are in our neck of the woods to join us. It is hosted by the Sentia Group in Lochristi.

TechNine March Updates for IT Pro's

Content

There are 3 sessions.The first session, Micha Wets – Enterprise Mobility MVP, introduces us all to Azure Policy Guest Configurations. What can this do today and what does it mean for Windows GPO configurations in the long run? Can they co-exist? Come find out!

Microsoft launched the new EDGE chromium-based browser. Erik Moreau – Windows Insider MVP will deliver his insights about this great new browser in the second session. I hope he has some good news about one of my main complaints about it. What is that? Attend and you’ll find out.

The 3rd session is still a mystery … TechNine likes to have a last moment surprise.

Practicalities

Schedule

  • 18h00: Welcome & Food
  • 18h30: Azure Policy Guest Configurations: The end of Windows GPO? – Micha Wets
  • 19h15: The New Edge – Erik Moreau
  • 20h00: Break
  • 20h15: To be announced, stay tuned!
  • 21h00: Networking and Questions
  • 21h30: End

When: Tuesday, March 3rd, 18:00 sharp

Location: Sentia, Denen 157, 9080 Lochristi, Belgium

Join us by registering here: March Updates for IT Pro’s

Veeam File Share backups and knowledge worker data

Introduction

Today I focus on Veeam File Share backups and knowledge worker data testing. In Veeam NAS and File Share Backups did my 1st testing with the RTM bits of Veeam Backup & Replication V10 File Share backup options. Those tests were focused on a pain point I encounter often in environments with lots of large files: being able to back them up at all! Some examples are medical imaging, insurance, and GIS, remote imaging (satellite images, Aerial photography, LIDAR, Mobile mapping, …).

The amount of data created has skyrocketed driven by not only need but the advances in technology. These technologies deliver ever-better quality images, are more and more affordable, and are ever more applicable in an expanding variety of business cases. This means such data is an important use case. Anyway for those use cases, things are looking good.

But what about Veeam File Share backups and knowledge worker data? Those millions of files in hundreds of thousands of folders. Well, in this blog post I share some results of testing with that data.

Veeam File Share backups and knowledge worker data

For this test we use a 2 TB volume with 1.87TB of knowledge worker data. This resides on a 2 TB LUN, formatted with NTFS and a unit allocation size of 4K.

1.87 TB of knowlegde worker dataon NTFS

The data consists of 2,637,652 files in almost 196,420 folders. The content is real-life accumulated data over many years. It contains a wide variety of file types such as office, text, zip, image, .pdf, .dbf, movie, etc. files of various sizes. This data was not generated artificially. All servers are Windows Server 2019. The backup repository was formatted with ReFS (64K allocation unit size).

Backup test

We back it up with the file server object from an all-flash source to an all-flash target. There is a dedicated 10Gbps backup network in the lab. As we did not have a separate spare lab node we configured the cache on local SSD on the repository. I set the backup I/O control for faster backup. We wanted to see what we could get out of this setup.

Below are the results.

Veeam File Share backups and knowledge worker data
45:28 minutes to backup 1.87 TB of knowledge worker data. I like it.

If you look at the back-up image above you see that the source was the bottleneck. As we are going for maximum speed we are hammering the CPU cores quite a bit. The screenshot below makes this crystal clear.

We have plenty of CPU cores in the lab on our backup source and we put them to work.
The CPU core load on the backup target is far less.

This begs the question if using the file share option would not be a better choice. We can then leverage SMB Direct. This could help save CPU cycles. With SMB Multichannel we can leverage two 10Gbps NICs. So We will repeat this test with the same data. Once with a file share on a stand-alone file server and once with a high available general-purpose file share with continuous availability turned on. This will allow us to compare the File Server versus File Share approach. Continuous availability has an impact on performance and I would also like to see how profound that is with backups. But all that will be for a future blog post.

Restore test

The ability to restore data fast is paramount. It is even mission-critical in certain scenarios. Medical images needed for consultations and (surgical) procedures for example.

So we also put this to the test. Note that we chose to restore all data to a new LUN. This is to mimic the catastrophical loss of the orginal LUN and a recovery to a new one.

The restore takes longer than the backup for the same amount of data, the restore speed is typically slower for large amounts of smaller files.

Below you will find a screenshot from the task manager on both the repository as well as the file server during the restore.

The repository server from where we are restoring the data
The file server where the backup is being restored completely on a new LUN. Note the peak throughput of 6.4 Gbps.

Mind you, this varies a lot and when it hits small files the throughput slows down while the cores load rises

The file server during the restore is having to work the hardest when it has to deal with the least efficient files.

Conclusion

For now, with variable data and lots of small files, it looks that restores take 2.5 to 3 times as long as backups with office worker data. We’ll do more testing with different data. With large image files, the difference is a lot less from our early testing. For now, this gives you a first look at our results with Veeam File Share backups and knowledge worker data As always, test for your self and test multiple scenarios. Your mileage will vary and you have to do your own due diligence. These lab tests are the beginning of mine, just to get a feel for what I can achieve. If you want to learn more about Veeam Backup & Replication go here.Thank you for reading.

Veeam NAS and File Share Backups

Introduction

Veeam NAS and File Share Backups are a new capability in Veeam Backup & Replication V10. We can now backup SMB, NFS shares as well as file server sources. This means it covers Linux and Windows files servers and shares. It can also backup also many NAS devices. There are many of those in both the SME and enterprise market. I know it is fashionable to state that file servers are dead, But that is like saying e-mail is dead. Yes, right until the moment you kill their mailbox. At that moment it is mission critical again.

My first test results with the RTM bits are so good I doing this quick publish to share them with you.

Early testing of Veeam NAS and File Share Backups

As a Veeam Vanguard I got access to the Veeam Backup & Replication V10 RTM bits so I decided to give it a go in some of our proving grounds.

I tested a Windows File Server, a Windows File share and a General Purpose File Share with continuous availability on a 2 node cluster. All operating systems run Windows Server 2019, fully patched at the time of writing.

Veeam NAS and File Share Backups
These are your 3 major options and they cover 99% of file share uses cases out there.

Windows File Server

This is the preferred method if you can use it. That is if you have a Windows or Linux server as opposed to an appliance. The speeds are great and I am flirting with the 10Gbps limits of the NIC. As this is pure TCP it does not leverage SMB Multichannnel or SMB Direct.

9.9Gbps, 800-900MB/s, what is not to like about such early test results?

Windows File Share

With a NAS you might not have the option to leverage the file server object. No worries, we than use the File Share. If it is SMB 3 you can even leverage VSS if the NAS supports it. It might have the added benefit that you can add more file share proxies to do the initial full backup if so required. With File Server you are limited to itself. But it all depends on what the source can deliver and the target ingest.

Veeam NAS and File Share Backups
A file share on Windows Server 2019

Note that we use a SMB 3 file share here. With a properly configured network you can leverage SMB Direct and SMB Multichannel with this.

RDMA and Multichannel – gotta love it.

General Purpose File Share with continuous availability on a 2 node cluster

This one is important to me. I normally only deploy General Purpose File Shares with continuous availability anymore where applicable and possible. SMB 3 haves given us many gifts and I like to leverage them. The ease of maintenance it offers is too good not to use it when possible. Can you say office hours patching of file servers?

So here is a screen shot of a Backup of a General Purpose File Shares with continuous availability where is initiate a fail over of the file share. That explains for the dip in the throughput, but the backup keeps running. Awesome!

Veeam NAS and File Share Backups
Our backup source fails over but the backup keeps running. That is what we like to see. All this also leverages SMB 3 Direct and Multichannel by the way.

Restores

Backups are cool but restores rule. So to finish up this round of testing we share a restore. Not bad, not bad at all. 221.9 GB restored in 5.33 minutes.

Speedy restores are always welcome

More testing to follow

I will do more testing in the future. This will include small office files in large quantities. These early tests are more focused on large image data such as satellite, aerial photography and mobile mapping images. An important use case, hence our early testing focus.

For an overview of Veeam NAS and file share backups as well as the details take a look here for a presentation on the subject by one and only Michael Cade at TFD20.

Conclusion

The Veeam NAS and file share backups in Backup & Replication V10 are delivering great results right from the start. I am happy to see this capability arrive. Without any doubt the only remark I have is that they should have done this sooner. But today, it is here and I am nothing but happy about this.

There are lot of details to this but that will be for later content.