Thursday, 31 March 2016

The Pluralsight Mobile App Sucks

There is a request to allow offline storage of courses on the SD card but it's been active for over two years now.

This can't be an insurmountable problem as quite a few apps quite happily allow this:


In any case, yesterday after I could not update several apps for lack of space for about the umpteenth time, I thought, enough is enough and I endeavored to find a solution.

I first wrote a small script in PowerShell that called Handle.exe on a loop, to get the list of files being opened by the pluralsight desktop app and wrote that list of files to a file:

if (-not (test-path 'handle.exe'))
{
 write-host "Can't find handle.exe"
 break;
}

$files= @()

while($true) 
{
 $video =.\Handle.exe -p pluralsight | Select-String mp4
 
 if ($video)
 {
  $path = $video.Line.Split(')')[1] 

  if (-not($files.Contains($path)))
  {
   Write-Host $path
   Write-Host "Files Added So far: " + $($files.Count)
   $files+=$path
   $files > file.txt
  }
 }
}

This required me to click on each chapter (video) of the course to get it to be logged by handle, which while a little bit annoying I thought could be automated somehow with CodedUI or AutoHotKey but the former was not very reliable and I didn't have the patience for the latter.

The above script worked on the first course I tried but it was a bit hit and miss on the second and after a lot of playing about I realized that for small videos it would simply not work, not sure why that is, is it reading the file quicker than the loop loops?  I've not stopped to do the maths but it seems odd in any case.

The second part of the process was the script below, which moves the files to a new folder so that they can be copied safely across to the phone/tablet:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$files = Get-Content $sourceFile
$counter = 0 

foreach($file in $files)
{ 
 $dest = $(join-path $destination $counter) + ".mp4"
 Copy-Item $file.Trim() $dest
 $counter++
}

If the first script grabbed all files, then this script worked fine, but like I said, it didn't always do it, it seemed not to work for small files.

So, I did a bit more googleing, alas, it seems that there is no strace equivalent in Windows and then when I was about to give up*, I hit upon Process Monitor, which does exactly what I needed, in other words, and among other things, it can list the files that an application opens. In order to do this, all that is needed is the correct filter and a little bit more of PowerShell scripting.

Open Process Monitor -> Filter and set the following filters (in green):


In reality, it's probably enough to do the path ends with mp4 filter as no operation seems to generate a single entry for a file read, which means that some processing will be needed, but the QueryNetworkOpenInformationFile Operation seems to generate the fewer hits so I stuck with that.

Note, sometimes it simply doesn't register anything, in which case, just restart Process Monitor.

With this running, I can now go through all the chapters of the course that I want to watch on my mobile and I get something like this:

Now it's time to export the results to a CSV file, which you can do by simply pressing CTRL + S

Finally, it's time to process the file that I've just saved, for which I slightly modified the second PowerShell script:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$source = Import-CSV $sourceFile
$files=@()
$source.Path | % { if (!$files.Contains($_)){$files+=$_}}

$counter = 0 

foreach($file in $files)
{
 if ($counter -lt 10) { $prefix= "0" + $counter} else {$prefix=$counter}
 $dest = $(join-path $destination $prefix) + ".mp4"
 Copy-Item $file.Trim() $dest
 $counter++
}

The above script is fine, but I'd rather have a single file than loads of files, so I use ffmpeg to concatenate the clips:
param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][string]$destFile, [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$ffmpeg)

$source = Import-CSV $sourceFile
$files=@()
$source.Path | % { if (!$files.Contains("file '" + $_.Replace('\','\\') + "'")){$files+="file '" + $_.Replace('\','\\') + "'"}}

$fileList = (Get-Date).Ticks

Set-Content -Path $fileList -Value $files

$x= $ffmpeg + " -f concat -i $filelist -codec copy $destFile"

Invoke-Expression $x

Remove-Item $fileList
Remove-Item $sourceFile -Confirm
Using this technique is actually quite flexible I can create individual videos for each module if a single video for the whole course is too unwieldy.

Unfortunately, this last step doesn't appear to work too well on new pluralsight videos, which is annoying as there doesn't appear to be a free Video Player for Android that does all of these:
  • Play all files from a directory.
  • Play at up 1.5x speed.
  • Allow the video to fill the screen.
So I say that there is no free video player that does this at the moment for Android.

Saturday, 27 February 2016

Collating and Plotting Perfmon data

We've been using Perfmon to collect information about the various apps we have for quite a while now and it was always a massive pain to present the information in a meaningful manner.

The problem is that each file contains a day's worth of data, I realize that it's possible to keep a counter for longer but the comparison problem remains the same and we wanted to be able to compare performance counters on different days.

I thought I would try to use Python and the pandas library for this.

I installed the pandas library by installing Anaconda, in particular I'm using version 2.3 for Python 2.7, yes it's all a bit confusing and no, I'm using Visual Studio but Spyder, which isn't' great but mostly works

The script below is very much tailor made to my current needs, so there are few quirks:
  1. It assumes all files will be full day files at 15 seconds intervals.
  2. It assumes that all files will have the same counters in the same order. It should be relatively trivial to change the script to take a config file with the counters needed.
  3. It calculates a total CPU usage of all processes except SQL Server by subtracting SQL server usage from total cpu usage.
There are several command line options that should, hopefully, be self explanatory

Without further ado here's the script:

Needs a bit more work - 
#way to limit the number of days to be plotted, 
#explore different plots perhaps for different counters
#Actually get the files from the servers

from os import listdir, mkdir
from os.path import isfile, join, exists
import pandas as pd
import matplotlib.pyplot as plt
import re
import sys

def RenameColumns (current, fileName):
    current = current[:-1]
    current.append(fileName)
    return current
    
def Process (allFiles, numberToPlot, category, currentFile):
    ##Add the date column. This will not work if different files start at different times and have different intervals
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #We only need the time, not the date.
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file number
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        df = df.join(allFiles[fi][1][allFiles[fi][1].columns[counter+1]])
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess.
  #I'm sure there are better ways of dealing with missing values than this
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x ==' ' else x ).astype(float)
    #Return a tuple with name and Dataframe.
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + category, df)       


def GetCPUUsageRest (allFiles, numberToPlot,numberOfCPUs,  currentFile):
     #Is this really needed...Test you say? ...    
    numberOfCPUs = float(numberOfCPUs)
     ##Add the date column
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #Get the time
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        #Hardcoding FTW because 
        total= pd.DataFrame(DataFrames[0][1])
        sql = pd.DataFrame(DataFrames[25][1])
        sql[fileName] = sql[fileName].apply (lambda x:x/numberOfCPUs)
        df = df.join(pd.DataFrame(total[fileName] - sql[fileName]))
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x < 0 else x ).astype(float)
    #Return a tuple with file name and Dataframe
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + "Rest-Process % CPU" , df)      
    
#These are the counters used. The files need to contain them all otherwise the plots will be all wrong
#I suppose that we could load a list of these from a config file to accomodate changes rather than this
#hard coding.    
counters=[]
counters.append('Processor(_Total)-% Processor Time')
counters.append('Memory-Available KBytes')
counters.append('.NET CLR Exceptions(_Global_)-# of Exceps Thrown  per  sec')
counters.append('.NET CLR Memory(_Global_)-% Time in GC')
counters.append('ASP.NET-Application Restarts')
counters.append('ASP.NET-Request Wait Time')
counters.append('ASP.NET-Requests Queued')
counters.append('ASP.NET-Request Execution Time')
counters.append('ASP.NET Applications(__Total__)-Requests per Sec')
counters.append('Memory-Pages per sec')
counters.append('PhysicalDisk(_Total)-% Disk Read Time')
counters.append('System-Processor Queue Length')
counters.append('Web Service(_Total)-Get Requests per sec')
counters.append('Web Service(_Total)-Post Requests per sec')
counters.append('Web Service(_Total)-Current Connections')
counters.append('SQLServer-Access Methods-Page Splits per sec')
counters.append('SQLServer-Buffer Manager-Buffer cache hit ratio')
counters.append('SQLServer-Buffer Manager-Page life expectancy')
counters.append('SQLServer-Buffer Manager-Checkpoint pages per sec')
counters.append('SQLServer-General Statistics-User Connections')
counters.append('SQLServer-General Statistics-Processes blocked')
counters.append('SQLServer-SQL Statistics-Batch Requests per sec')
counters.append('SQLServer-SQL Statistics-SQL Compilations per sec')
counters.append('SQLServer-SQL Statistics-SQL Re-Compilations per sec')
counters.append('Network Interface-Bytes Total per sec')
counters.append('SQLServer-Process % CPU')

#This is regex for dates in the format yyyyMMdd. It doesn't quite validate as it would happily allow a day of 39
regexDate = '(20\d{2}[0-2]\d{1}[0-3]\d{1})'
#Set plotting style
plt.style.use('ggplot')

cmdargs = str(sys.argv)

if len (sys.argv) != 6 :
    print "This script takes five arguments"
    print "First Argument should be path where csv files to be processed are"
    print "Second Argument should be path where plots will be stored"
    print "Third Argument should be server name"
    print "Forth Argument should be number of Days to Plot"
    print "Fifth Argument should be number of CPUs"
    sys.exit(0)

path = sys.argv[1].rstrip('\\"')
dest = sys.argv[2]
server = sys.argv[3]
numberOfDaysToPlot = int(sys.argv[4])
numberOfCPUs = int(sys.argv[5])

    
if not exists(dest):
    mkdir(dest)

DataFrames = []
allFiles=[]
#Read the files for the input path. 
files = [f for f in listdir(path) if isfile(join(path,f)) and f.endswith('csv')]
for f in files:    
    name = re.search(regexDate, f).group()
    allFiles.append((name, pd.read_csv(join(path,f))))

remainingFiles = len(files)
#It looks like perfmon adds an extra column at the end of the file, which isn't a counter, hence the - 2
for counter in range(len(allFiles[0][1].columns)-2):
    currentFile = 0 
    while remainingFiles > 0:
        DataFrames.append(Process(allFiles, numberOfDaysToPlot, counters[counter], currentFile))
        #if we've reached the last counter, then we can look at the rest
        if counter == len(allFiles[0][1].columns)-3:
            DataFrames.append(GetCPUUsageRest(allFiles, numberOfDaysToPlot, numberOfCPUs, currentFile))
        remainingFiles = remainingFiles - numberOfDaysToPlot
        currentFile = currentFile + numberOfDaysToPlot
    remainingFiles = len(files)

CPUylim = ()     
#All plotting is done here.
for d in DataFrames:
    p=d[1].plot(x='date', title=d[0])
    #Get Y Axis Limit for Total CPU
    if d[0].find('Processor(_Total)-% Processor Time') != -1:
        CPUylim = p.get_ylim()        
    if d[0].find('SQLServer-Process % CPU') != -1:
        p.set_ylim( CPUylim)
    if d[0].find('Rest-Process % CPU') != -1:
        p.set_ylim(CPUylim)   
    #These two set the x ticks and labels make sure that they roughly match up.
    p.set_xticks([750.0,1500.0,2250.0,3000.0,3750.0,4500.0, 5250.0])
    p.set_xticklabels(['03:07:30','06:15:00','09:22:30','12:30:00','15:37:30','18:45:00','21:52:31'])
    fig = p.get_figure()
    fig.set_size_inches(21,13)
    fig.savefig(join (dest, d[0] + ".png"))
    plt.close()
A sample plot generated

Monday, 15 February 2016

Can't afford a certificate

Today somebody at work told me to log in to our gotomypc account and I manually typed the address: https://www.gotomypc.co.uk/


Clearly the UK is not an important enough market to merit a proper cert or a SAN.


This is the certificate for the .com website:


Wednesday, 3 February 2016

MSDN Subscription - A Very British Rip-Off

So a few days back I was considering whether we should take the plunge and fork out for an MSDN subscription for all our developers, there is five of us and given where we want to be it made sense.

This was a cost that had been budgeted for so, in theory, this should not be a problem, but as with all large purchases, and for a small firm, 5 MSDN subscriptions is a Large purchase, which meant that we had to get approval from above so I went to this site to confirm prices.


5 x $1199 even at silly exchange rates, should be less than £5K, so I thought, this would be an easy sell as it was less than we had budgeted, boy was I wrong.

If you are in the UK, you can see for yourself here, which shows you this:


What????? 

The exchange rate used by Microsoft is $1 = £1.01, rather than $1 = £0.68, so close to 50% surcharge. 

To put it another way, if we converted back to dollars, the price would be ~ $1780 rather than $1199.

I really like to hear an explanation from Microsoft, it certainly can't be the taxes the have to pay in this country, as the sales will probably be billed to the Irish subsidiary.

For the record this is the exchange rate today, according to Google:



and for the really lazy: 


I know there are resellers out there that do a cheaper price, it still does not change the fact that Microsoft itself applies an almost 50% surcharge to UK developers.

Sunday, 24 January 2016

Migrating from Mercurial Kiln to Visual Studio Online (Git)

So we migrated all of our Mercury repositories, hosted in Kiln, to Visual Studio Online last week and this is the procedure we followed.

We only had 3 separate repositories and they are relatively small, so we deemed it unnecessary to try a more scalable approach. 

All the command line commands are run from a Developer Command Prompt for VS 2015.

1. Clone Mercury Repository to be migrated to a new directory.
hg clone <repo url>
2. Create new Harmony Repository
a. Navigate to https://<yourcompany>.kilnhg.com/Code/Repositories
b. Click on New Repository

It's very important that you select a Harmony Repository, otherwise this process will not work. Forking or Branching a Mercury only repository will not work either, as it simple creates a new Mercury repository.
3. Push to new Harmony Repository
 a. Edit  the .hg/hgrc file and change the path the new repository url
[paths]
default = https://<>.kilnhg.com/Code/Repositories/Group/Test_Migration
b. hg push  
Once this is has finished the Test_Migration repository is a Git & Mercury repository of the initial Mercury only repo, so it's relatively simple from this point onward.
4. Clone Harmony Repository as a Git Repository
git clone <https://<>.kilnhg.com/Code/Repositories/Group/Test_Migration.git>
5. Create new Repository in VSO
a. Navigate to Project
b. Click on Code
Once created you can click on Explorer which will take you to this page

6. Push to VSO
git remote remove origin
git remote add origin https://<yourcompany>.visualstudio.com/DefaultCollection/<project>/_git/<repo>
git push -u origin --all

Sunday, 10 January 2016

Cloning Reddit to learn ASP.NET 5 - Part 6

In the last post in the series, we used a DTO to minimize the transfered data but we retrieved all that data from the database, shouldn't we have got just what we wanted?

In short the answer is yes, again the potential savings need to be carefully considered as the code might be more cluttered.

This is the query that gets run against the database:

SELECT [x].[SubRedditId], [x].[IsBanned], [x].[IsDefault], [x].[IsPublic], [x].[Name]
FROM [SubReddit] AS [x]
WHERE ([x].[IsDefault] = 1 AND [x].[IsPublic] = 1) AND [x].[IsBanned] = 0
You might be tempted to modify the Repository method like this:
public IEnumerable<SubRedditDTO> GetDefaultSubReddits()
{
            return ctx.SubReddits
                .Where(x => x.IsDefault && x.IsPublic && !x.IsBanned)
                .Cast<SubRedditDTO>();       
}
If you do this, you'll get this error: System.InvalidOperationException:
No coercion operator is defined between types 'Reddit.Models.SubReddit' and 'Reddit.Models.SubRedditDTO'.
We could use AutoMapper here but it seems a bit clunky when you're trying to reduce object size as the definition needs to be a bit like this:
          Mapper.Initialize(config =>
            {
                config.CreateMap<SubReddit, SubRedditDTO>()
                .ForSourceMember(x => x.IsBanned, o => o.Ignore());               
            });
In all honesty I've not read enough about AutoMapping, DTOs and ViewModels to be able to advise on best practices, but in this case it does seem a bit clunky, so instead I tried the following:
       public IEnumerable<SubRedditDTO> GetDefaultSubReddits()
        {
            return ctx.SubReddits
                .Where(x => x.IsDefault && x.IsPublic && !x.IsBanned)
                .Select(x => new SubRedditDTO() { Name=x.Name });
        }
This actually achieves what we wanted, at least in theory, which is reduce the load on the database. I realize that DB performance is a lot more nuanced than I'm presenting it here, but as a rule of thumb the less data retrieved from the database the better. This is the query that is executed against the database:
SELECT [x].[Name]
FROM [SubReddit] AS [x]
WHERE ([x].[IsDefault] = 1 AND [x].[IsPublic] = 1) AND [x].[IsBanned] = 0
This should be better from a DB performance point of view, but how much, it's hard to say.

The last thing I'll tackle in this post is the lack of links on the front page. You might be tempted to simply change the Index.cshtml page to this:

@{
    ViewData["Title"] = "Reddit";
}
@section scripts {
    <script src="~/lib/angular/angular.js"></script>
    <script src="~/js/redditapp.js"></script>
}
<div ng-app="redditApp">
    <div ng-controller="SubRedditListCtrl">
        <ul class="list-inline">
            <li ng-repeat="subReddit in subReddits">
                <a href="http://reddit.com/r/{{subReddit.Name}}">{{subReddit.Name}}</a>                      
            </li>
        </ul>
    </div>
    <hr />
</div>
Unfortunately, that will not always work as expected, all links will not have the subreddit until angular has substituted the value of subReddit.Name, instead we need to use a angular directive:

<a ng-href="http://reddit.com/r/{{subReddit.Name}}">{{subReddit.Name}}</a>  

Sunday, 3 January 2016

Cloning Reddit to learn ASP.NET 5 - Part 5

In the last post we finally started to do something to the front end, namely the SubReddit bar on top, but the values were hard coded and they weren't even links.

Let's try to remedy that.

There are a few things we need to do:
  • How do we identify Default SubReddits?
  • How do we get and display said SubReddits?
The first one is easy, we add a new property to our SubReddit class, which now looks like this:
using System.Collections.Generic;

namespace Reddit.Models
{
    public class SubReddit
    {
        public int SubRedditId { get; set; }
        public string Name { get; set; }
        public bool IsPublic { get; set; }
        public List<Post> Posts { get; set; }
        public List<RedditUser> Users { get; set; }
        public bool IsBanned { get; set; }
        public bool IsDefault { get; set; }
    }
}

There are other ways of doing this, say, have a default SubReddit table an add the default SubReddits there, but we'll do it this way until I change my mind.

We need to run now through the Migrations rigmarole again, so from ..\src\Reddit run the following commands:
dnx ef Migrations add isDefaultSubReddit
dnx ef database update
I've manually seeded the SubReddit table with a few records, both default and not default (I will discuss how to seed it from code in a later post) and now we can edit redditApp.js to this:
(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', ['$scope', '$http', function($scope, $http) {
  $http.get("/api/SubReddit").then(function(data){
  $scope.subReddits =data.data ;
})}
])
  
})();

In order to get around issues with minification, we inject the dependencies as strings which will not be minified.

There are several issues with this approach as we're getting more data than we need in two senses:

  • We're getting all SubReddits, we could filter them but that would mean retrieving all the data from the server and then filtering them, which is as inefficient as it sounds. 
  • Furthermore, we're retrieving all properties of the SubReddit objects when we only need the name.
Let's deal with the first of these issues. We add a new method called GetDefaultSubReddits to the Repository, make sure you add it to the Interface as well:

        public IEnumerable<SubReddit> GetDefaultReddits()
        {
            return ctx.SubReddits.Where(x => x.IsDefault && x.IsPublic && !x.IsBanned);
        }
And we also add a call to this method from the Controller:
.        [HttpGet("[action]")]
        public IEnumerable<SubReddit> GetDefault()
        {
            return repository.GetDefaultSubReddits();
        }
Note that I'm using an action here, which is effectively the name of the method
Now we modify the redditApp.js
(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', ['$scope', '$http', function($scope, $http) {
  $http.get("/api/SubReddit/GetDefault").then(function(data){
  $scope.subReddits =data.data ;
})}
])
  
})();
At this point we've solved issue number one but we're still sending quite a bit of unnecessary data up and down the wire so let's build a DTO that contains just the data that we need, which in this case is simply the Name. There are other ways of achieving this, for instance we could use an OData query to simply retrieve the necessary data.

This is the SubRedditDTO

namespace Reddit.Models
{
    public class SubRedditDTO
    {
        public string Name { get; set; }
    }
}
Modified GetDefault Method, I'm not sure whether it's a better idea to transform to DTO object here or on the Repository. I guess it depends on usage, in other words, if the Repository method will be used somewhere else it makes sense to do the transformation here.
        [HttpGet("[action]")]
        public IEnumerable<SubRedditDTO> GetDefault()
        {
            return repository.GetDefaultSubReddits().Select(x => new SubRedditDTO() { Name = x.Name });
        }
This is the reponse from the server now
[{"Name":"Gadgets"},{"Name":"Sports"},{"Name":"Gaming"},{"Name":"Pics"},{"Name":"WorldNews"},{"Name":"Videos"},{"Name":"AskReddit"},{"Name":"Aww"}]
This is before, when we where not using a DTO.
[{"SubRedditId":3,"Name":"Gadgets","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":4,"Name":"Sports","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":5,"Name":"Gaming","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":6,"Name":"Pics","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":7,"Name":"WorldNews","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":8,"Name":"Videos","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":9,"Name":"AskReddit","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":10,"Name":"Aww","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true}]
This might seem like a paltry gain but little savings like this can quickly add up.

Saturday, 2 January 2016

Cloning Reddit to learn ASP.NET 5 - Part 4

In the previous post we ended up again with a broken, although partially this time, application let's try to fix it.

The problem was that it could not resolve the service type for IRedditRepository as we were not telling how to inject the dependency properly, which is done on the ConfigureServices method on Startup.cs:

public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();
            services.AddEntityFramework()
                .AddSqlServer()
                .AddDbContext<RedditContext>();

            services.AddScoped<IRedditRepository, RedditRepository>();
        }
AddScoped will use the same instance for each request, could have used AddSingleton or AddInstance.

We try running this again and we see this:


So it's working but there is nothing there, we'll have a look at this later on.

Now let's try to look at the homepage, which at the moment looks like this.

This doesn't look like Reddit, so let's do a bit of editing.

Delete the Contact.cshtml and About.cshtml files, in the Home folder

Edit Index.cshtml so that it looks like this:

@{
    ViewData["Title"] = "Reddit";
}
This is the _Layout.cshtml file, which can be thought of as a master page or template page. I've removed pretty much everything from it that might be rendered.
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>@ViewData["Title"] - Reddit</title>

    <environment names="Development">
        <link rel="stylesheet" href="~/lib/bootstrap/dist/css/bootstrap.css" />
        <link rel="stylesheet" href="~/css/site.css" />
    </environment>
    <environment names="Staging,Production">
        <link rel="stylesheet" href="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.5/css/bootstrap.min.css"
              asp-fallback-href="~/lib/bootstrap/dist/css/bootstrap.min.css"
              asp-fallback-test-class="sr-only" asp-fallback-test-property="position" asp-fallback-test-value="absolute" />
        <link rel="stylesheet" href="~/css/site.min.css" asp-append-version="true" />
    </environment>
</head>
<body>
    <div>
        @RenderBody()       
    </div>

    <environment names="Development">
        <script src="~/lib/jquery/dist/jquery.js"></script>
        <script src="~/lib/bootstrap/dist/js/bootstrap.js"></script>
        <script src="~/js/site.js" asp-append-version="true"></script>
    </environment>
    <environment names="Staging,Production">
        <script src="https://ajax.aspnetcdn.com/ajax/jquery/jquery-2.1.4.min.js"
                asp-fallback-src="~/lib/jquery/dist/jquery.min.js"
                asp-fallback-test="window.jQuery">
        </script>
        <script src="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.5/bootstrap.min.js"
                asp-fallback-src="~/lib/bootstrap/dist/js/bootstrap.min.js"
                asp-fallback-test="window.jQuery && window.jQuery.fn && window.jQuery.fn.modal">
        </script>
        <script src="~/js/site.min.js" asp-append-version="true"></script>
    </environment>

    @RenderSection("scripts", required: false)
</body>
</html>
The end result of all this should be an empty page.

I've left RenderBody and RenderSection method calls intact as they will come in handy later. In essence, they are used to merge in the content from the actual pages and to load the scripts needed respectively.

Since this is a modern app I'm going to use AngularJS, which is probably past it's prime now but there you go.

The simplest thing to do this is by using the manage bower packages:


This will display this page:


This is the result on the bower.json file, which should be noted is not shown by visual studio but it's there on the src\Reddit folder:

{
  "name": "ASP.NET",
  "private": true,
  "dependencies": {
    "bootstrap": "3.3.5",
    "jquery": "2.1.4",
    "jquery-validation": "1.14.0",
    "jquery-validation-unobtrusive": "3.2.4",
    "angular": "1.4.6"
  }
}
So let's add Angular to Reddit

I've added a new file called redditApp.js to wwwroot/js, which contains the following:
(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', function($scope) {
  $scope.subReddits = [
    {'name': 'Gadgets'},    
    {'name': 'Sports'},
    {'name': 'Gaming'},
    {'name': 'Pics'},
  ];
});
  
})();

I've hard coded a few subreddits to have something to display but we will soon get them from the database. In essence, the above is a self executing function containing an extremely simple angular controller that has an array of objects.

I have modified the Index.cshtml like this:

@{
    ViewData["Title"] = "Reddit";
}
@section scripts {

    <script src="~/lib/angular/angular.js"></script>
    <script src="~/js/redditapp.js"></script>
}
<div ng-app="redditApp">
    <div ng-controller="SubRedditListCtrl">
        <ul class="list-inline">
            <li ng-repeat="subReddit in subReddits">
                <span>{{subReddit.name}}</span>
            </li>
        </ul>
    </div>
</div>

Note the angular directives? ng-app, ng-controller and ng-repeat and also note the angular bingind {{subReddit.name}}

This is what Reddit looks like now:

Friday, 1 January 2016

Cloning Reddit to learn ASP.NET 5 - Part 3

In the last post we applied a database migration but nothing occurred.

The reason for this was that the RedditContext class was not tracking any of the objects nor where the relationships specified on the OnModelCreating.

So I've modified the RedditContext class

using Microsoft.Data.Entity;

namespace Reddit.Models
{
    public class RedditContext : DbContext
    {
        public RedditContext()
        {
            //Database.EnsureCreated();
        }

        public DbSet<RedditUser> RedditUsers { get; set; }
        public DbSet<UserDetails> UserDetails { get; set; }
        public DbSet<SubReddit> SubReddits { get; set; }
        public DbSet<Post> Posts { get; set; }
        public DbSet<Comment> Comments { get; set; }
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            var connection = Startup.Configuration["Data:RedditContextConnection"];
            optionsBuilder.UseSqlServer(connection);
            base.OnConfiguring(optionsBuilder);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<RedditUser>()
                .HasKey(x => x.RedditUserId);

            modelBuilder.Entity<UserDetails>()
                .HasKey(x => x.UserId);

            modelBuilder.Entity<RedditUser>()
                .HasOne(x => x.UserDetails)
                .WithOne(x => x.User);

            base.OnModelCreating(modelBuilder);
        }
    }
}

In order to recover from this mistake, we want to apply the new migration, but the Core migration contains nothing so I will first unapply it
dnx ef database update Start
dnx ef migrations remove
dnx ef migrations add Core
dnx ef database update
This time it worked .... hurray


At this point, it's worth considering how will this be deployed to a test environment and the answer is pretty simple. A migration script can be created with the following command:
dnx ef migrations script
This creates a Sql script that can be used to create the database.

It's time to have another look at our repository
Let's start with the IRedditRepository, where we now have a few methods and this time we'll try to implement some of them.
using System.Collections.Generic;

namespace Reddit.Models
{
    public interface IRedditRepository
    {
        UserDetails GetUserDetails();
        IEnumerable<SubReddit> GetSubscribedSubReddits(RedditUser user);
        IEnumerable<SubReddit> GetSubReddits();
        IEnumerable<Post> GetSubRedditPosts(SubReddit subreddit);
        IEnumerable<Comment> GetPostComments(Post post);
    }
}
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Linq;

namespace Reddit.Models
{
    public class RedditRepository : IRedditRepository
    {
        private readonly RedditContext ctx;
        private readonly ILogger<RedditRepository> logger;

        public RedditRepository(RedditContext ctx, ILogger<RedditRepository> logger)
        {
            this.ctx = ctx;
            this.logger = logger;
        }

        public IEnumerable<Comment> GetPostComments(Post post)
        {
            return ctx.Posts
               .Where(x => x.PostId == post.PostId)
               .Single()
               .Comments;
        }

        public IEnumerable<Post> GetSubRedditPosts(SubReddit subreddit)
        {
            return ctx.SubReddits
                .Where(x => x.SubRedditId == subreddit.SubRedditId)
                .Single()
                .Posts;
        }

        public IEnumerable<SubReddit> GetSubReddits()
        {
            return ctx.SubReddits.AsEnumerable();
        }

        public IEnumerable<SubReddit> GetSubscribedSubReddits(RedditUser user)
        {
            throw new NotImplementedException();
        }

        public UserDetails GetUserDetails()
        {
            throw new NotImplementedException();
        }
    }
}

Straight away we see that there are going to be problems as we lack paging capabilities, but we'll add those later

In any case, we'll need to expose these method somehow, there are potentially two options here depending on what we're trying to accomplish: 

We could use these methods from inside a Controller or We could create an API and we're definitely going for the latter as it's way cooler

Create a new folder called Api inside the Controllers folder and add a new  Web API controller class, called SubRedditController
using System.Collections.Generic;
using Microsoft.AspNet.Mvc;
using Reddit.Models;

// For more information on enabling Web API for empty projects, visit http://go.microsoft.com/fwlink/?LinkID=397860

namespace Reddit.Controllers.Api
{
    [Route("api/[controller]")]
    public class SubRedditController : Controller
    {
        private readonly IRedditRepository repository;

        public SubRedditController(IRedditRepository repository)
        {
            this.repository = repository;
        }

        // GET: api/values
        [HttpGet]
        public IEnumerable<SubReddit> Get()
        {
            return repository.GetSubReddits();
        }

    }
}

We can now build and debug the project. This allow us to navigate to http://localhost:<portnumber>/api/SubReddit

Unfortunately this does not work:


We'll look at the solution to this problem in the next post.

Thursday, 31 December 2015

Cloning Reddit to learn ASP.NET 5 - Part 2

In the previous post we got started in our quest to build a Reddit Clone. We ended up with a broken project so let's try to fix that.

The first thing is to add a UserDetails class and then link it to a RedditUser class, which will also need to be added to the Models folder.

UserDetails.cs
namespace Reddit.Models
{
    public class UserDetails
    {
        public int UserId { get; set; }
        public long LinkKarma { get; set; }
        public long CommentKarma { get; set; }
        public RedditUser User { get; set; }      
    }
}

RedditUser.cs
namespace Reddit.Models
{
    public class RedditUser
    {
        public int RedditUserId { get; set; }
        public string Nick { get; set; }
        public UserDetails UserDetails { get; set; }
    }
}

We will further modify the RedditUser class to allow users to login, that's the reason for the rather small class. If this were a real clone then we'd probably want to use Guids for the Keys.

At this point we should have a working project again, which can be verified by building and/or debugging it.

We now want to create the database with the RedditUser and UserDetails and the right relationships between them, namely a 1 to 1 relationship.

Firstly, modify the Startup.cs file so that the Configuration property is static, which will allow us access to this property without having to instantiate the class again and to add the EntityFramework service.
public static IConfigurationRoot Configuration { get; set; }

public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();
            services.AddEntityFramework()
                .AddSqlServer()
                .AddDbContext<RedditContext>();      
        }
Edit the appsettings.json file
 "Data": {
    "RedditContextConnection": "Server=(localdb)\\MSSQLLocalDB;Database=Reddit;Trusted_Connection=true;MultipleActiveResultSets=true"
  }
Finally let's edit the Context class (RedditContext.cs)
using Microsoft.Data.Entity;

namespace Reddit.Models
{
    public class RedditContext : DbContext
    {
        public RedditContext()
        {
            //Database.EnsureCreated();
        }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            var connection = Startup.Configuration["Data:RedditContextConnection"];
            optionsBuilder.UseSqlServer(connection);
            base.OnConfiguring(optionsBuilder);          
        } 

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<RedditUser>()
                .HasKey(x => x.RedditUserId);

            modelBuilder.Entity<UserDetails>()
                .HasKey(x => x.UserId);
            
            modelBuilder.Entity<RedditUser>()
                .HasOne(x => x.UserDetails)
                .WithOne(x => x.User);

            base.OnModelCreating(modelBuilder);
        }
    }
}

The OnConfiguring method sets up the connection String to the database.

The OnModelCreating method sets up the relationship between the tables in the database, it's worth mentioning here that I'm using the Fluent API to do this, but it's also possible to do it via data annotations.

I'm torn as to what the best way of doing this is, as both have advantages and disadvantages. I suspect that I will stick to the this syntax for the time being, even if I probably won't need any of the features that it provides over and above data annotations

I have added explicitly added the Primary Keys for each table, this is only necessary for UserDetails as the Key is different to the table name (I think) Now we can add create the database, from src\Reddit run the following command, which will create a Migration called Start:
 dnx ef migrations add Start 
Note that a new folder called Migrations will be created and populated with two files: A migration file, normally named DateTime_MigrationName and a model Snapshot.

When I first run this, it created the database. This was due to the commented out logic in the constructor, which wasn't commented out, once it was commented out, the expected behaviour resumed and we need to run the following command to apply the changes:
dnx ef database update
Let's add a few more models for SubReddit, Posts and Comments:

using System.Collections.Generic;

namespace Reddit.Models
{
    public class SubReddit
    {
        public int SubRedditId { get; set; }
        public string Name { get; set; }
        public bool IsPublic { get; set; }
        public List<Post> Posts { get; set; }
        public List<RedditUser> { get; set; }
        public bool IsBanned { get; set; }

    }
}
using System.Collections.Generic;

namespace Reddit.Models
{
    public class Post 
    {
        public int PostId { get; set; }
        public string Title { get; set; }
        public string Content { get; set; }
        public string Link { get; set; }
        public int UpVotes { get; set; }
        public int DownVotes { get; set; }
        public List<Comment> Comments { get; set; }
        public RedditUser Author { get; set; }
        public int AuthorId { get; set; }
        public SubReddit SubReddit { get; set; }
        public int SubRedditId { get; set; }


    }
}
namespace Reddit.Models
{
    public class Comment
    {
        public int CommentId { get; set; }
        public string Title { get; set; }
        public string Text { get; set; }
        public int UpVotes { get; set; }
        public int DownVotes { get; set; }
        public RedditUser Author { get; set; }
        public int AuthorId { get; set; }
    }
}
I have added Foreign Keys to Comment and Post to make querying easier. We create another migration called Core and apply it
dnx ef migrations add Core
dnx ef database update
Oddly nothing happens, why?