Beginnings in Golang and AWS – Part II – Uploading to S3

Introduction

With the preambling and prerequisities of Part I out of the way, we can now begin working on writing some code to allow us to upload the MP4 file to an S3 bucket.

In this post, we’ll cover the format of a Go package, how to add packages to an installation of Go, the import statement, and lastly how we go about parsing command line options.

It might not seem a great deal of code, but there’s quite a lot of concepts covered here that are essential to understanding how Go works. We’ll then be primed for the final part of the series on S3, which will cover the rest of the code, compiling, running, and using this program.

Uploading a File to S3

I’ll show the complete code first, and then break it down into parts.

Code Breakdown

Let’s go through the code and get a feel for what’s happen here.

VERY important! Go is case sensitive. Capitalization is treated differently, both from a name interpretation, and also operational point of view.

The Package Declaration

Every Go program consists of one or more packages. For a program to run (as opposed to being a resource for another program), it requires a main package.

Define Packages to be Used

Multiple packages exist for Go, both as part of a default installation, and also from the community.

The import statement tells Go what packages (and consequently resources such as functions and types) are available to the program. We can either do separate import statements, or group them together like above.

Go has a default package directory setting for packages not included in the default installation, from which it attempts to find the package (typically ~/go/src).

For example, github.com/aws/aws-sdk-go/aws, referred to in the import statement above, is located at the following location under my home directory:

When you want to use a resource in a package, such as a function or type, you need to refer to it including the package name. So if we wanted to use the Printf function within the fmt package with write a message, an example of this would be:

fmt.Printf(“No Hello World today”)

Define the Main Function

The entry point for a file to be executable (as opposed to solely a resource) is the main() function. The code executed within the function, represented by the dots, is enclosed within { and } braces.

Configure Command Line Parameter Parsing

When we execute this program from the command line, we want to include parameters which will define both the s3 bucket we want to upload to, and also the source file. The values need to be parsed and assigned to variables. To make it easier, we also want to provide some help text for people running the program.

Several things happen with the above code, so let’s go through them.

  • Both the bucket and filename variables are defined. Go normally requires a variable and its type to be pre-declared before it can be used. However, it is possible to create a variable with its type, and assign a value to it by using  :=  Quite simply what this does is leave it to the right of the operator to provide and type and value information. In this case, it is using the result of the String function in the flag package.
  • We use the flag package. The flag package has functions that allow us to parse the command line. We use flag.String to define a string flag with specified name, default value, and usage string. The return value is a reference to the memory address (aka pointer) which stores the value of the flag.
  • The Parse function is called. This carries out the processing of the command line parameters. This function sets the values at the memory location referred to by bucket and filename
  • It’s worthwhile mentioning that the output that will be generated if help is requested on our program, once compiled, is defined in this code as well. We’ll see in the last part on the S3 Uploader just exactly how this works.
  • You also might be wondering why the function name is capitalized. This is because in order for a resource in a package to be used by another, the initial letter must be a capital one. This marks it as “exportable”, allowing its use elsewhere.

Conclusion

In this post, we’ve covered a lot of topics, such as how we can use existing packages with our go program, how packages are stored locally, the effect that using lower and uppercase letters can have, the requirements for a program in go, and the import statement. We also began to delve in assignment by inference, pointers, flags  and how we can parse them.

Wit these out of the way, we’re primed and pretty much tickety boo for the final part of the series on S3, which will cover the rest of the code, introducing further concepts and syntax, including compiling, running, and using this program.

Thanks for reading!

Share

Beginnings in Golang and AWS – Part I

Part I – Introduction, Go, and Prerequisites

The series of blogs describe my journey learning Golang so far, with particular reference to AWS services. In them, I’ll be covering how I was able to achieve a task with AWS services with Go, but also what I learnt about Go at the same time. In addition to some services already mentioned in previous blogs, we’ll also be covering Lex, API Gateway, Lambda, and Alexa Skills.

Introduction

DSL’s aside, it’s been quite some time since I’ve endeavored to learn a new language in IT. Aside from a smattering of parsing through some Ruby and C# code on occasions and a bit of Python with BOTO3 for AWS stuff, it’d be fair to say the last one I learned to any level of depth was PowerShell.

Go

Already providing support for Node.js, Java, C# and Python with Lambda, AWS announced in January this year the addition of Go (short and more commonly used form of Golang). I’ve seen a lot of enthusiasm around the communities about Go, and with it marked as cross platform, fast, and of a relatively simple nature (sorta), I decided I’d give this a bash (no pun intended).

Whether it’s because I’m over-enthusiastic or (more likely) of a completely disorganized mind, my usual modus operandi involves skipping “hello world” learning completely and just diving in. Most certainly not a purist compliant approach…
I figured a good way to try this would be to adapt some of the PowerShell scripts I’ve written previously that feature here. Unoriginal for sure, but as it’s fresh in my mind, why not try with Golang to perform the creation of an SRT file?

Previously discussed in the PowerShell blogs, the task effectively comprises uploading a media file to an S3 bucket, creating a Transcribe job, obtaining the results, and then converting them.

As we’ll see later in this series of blogs though, additional options are available for how we carry some of these tasks out, making further automation and input sources possible.

For now though, we’ll get the pre-requisites out of the way.

Preparation

  • An existing AWS account. You can sign up for a Free Tier account here
  • Credentials and configuration are setup. See here for information on how to do this.
  • Go is installed (see below)
  • AWS SDK for Go is installed (see below)
  • You have cloned the repo for the project from either its source or your own fork
  • You’ve a ready-to-hand MP4 media file
  • You have a suitable development environment, such as VS Code

Install Go

Naturally, you are going to need Go installed. You can find your appropriate binary at the downloads page of golang.org. I’ll leave it to you to follow the specific installation instructions and any steps needed to fulfill requirements.

Install the AWS SDK for Go

You’ll also need the AWS SDK. Go has a really nice way to add packages to your environment, which is simply achieved by using the go get xxxxx, where xxxxx is the location of the package from the command line. Go handles the rest.

For installation of the AWS SDK for Go, simply use the following:

go get github.com/aws/aws-sdk-go/

Install VS Code (optional)

I currently use VS Code, which has some nice extensions to do tasks such as formatting, and automatic adding and removal of import commands. If you’re not currently using a development environment, or just simply fancy trying it out, you can obtain the binaries (available for Windows, OSX, and Linux) from here:

Add the Go Extention to VS Code

Follow the instructions and after installation is complete, launch VS Code. Add the extention ms-vscode.go, which provides features such as mentioned above within the development environment.

Conclusion

With the prerequisites now in place for us to use Go with AWS services, we can begin the process of putting together code which will help us achieve the tasks we’ve already mentioned.

In the next blog, we’ll dive into writing a script that will upload our media file to S3 using the SDK, and at the same time learning about a bit about Go.

Thanks for reading!

Share

When Marvin Gaye Met Amazon Transcribe & PowerShell – Automating Subtitle Creation – Part III

“I Heard it Through the Grape Vernon”

Part two of this series saw us put the code in place to allow us to upload the media file, create the transcription job, and download the results. With this complete, it’s time to move on to the next, and final, stage. That is, processing the JSON file, and creating an SRT file from it.

NB. You can find the code used in this blog, and additional documentation, at my Github project, aws-powershell-transcribe2srt:

Initially, we’ll read the contents of the json file into a variable and specifically use the items section, which contains the word-by-word breakdown. At the same time, some other variables are set with defaults.

Next, the Transcription variable needs to be processed. There’s several things that need to be taken into account:

  • The obvious one is that we need to parse through each item in the object, requiring a loop.
  • We also want to ensure that the number of words displayed per line does not exceed recommendations
  • In relation to the above, we also need to define the end time for each sequence the same as the last word.
  • Lastly, punctuation needs to be taken into account.

Outer Loop – Beginning

Using a while loop, the variable $strlen is set to zero. This variable will contain a count of the number of characters in the current sequence being processed.

Process the Start Time Attribute

Then, the time at which the word was said, start_time is read. In its original json format, this is as below.

However, the format for an SRT file of a time element consists of the number of hours, minutes, seconds, and milliseconds. All but the last of these use a fixed two character zero-padded digit format. The last uses three digits.

We’ll convert this string into the required style by first converting it to a timespan object, and then using string formatting to set it as required. Variables are also set for the subtitle text, the sequence number, and a flag to indicate that we are beginning the first line of this subtitle sequence. A variable is set for the end time too, and the process continues until 64 characters have been exceeded.

Inner Loop

An inner loop is also required, since we want the subtitles to be refreshed after two lines and with a maximum of 64 characters. Whether the item is a pronunciation (aka word) or punctuation needs to be taken into account, as well as setting the end time marker for the sequence when two lines have been occupied.

The type of item is read (pronunciation or punctuation), its content, and the length of the string tally is increased in line with this. Based on the type of item, the subtitle string is appended accordingly. When the length of the string exceeds 32 characters, and we are still on the first row, a new line character is added, and the variable indicator set to indicate that the subsequent content will be on the second line.

Outer Loop – End

When the inner loop is complete, it signifies a new sequence is ready. This part simply creates the appropriate representation of the sequence as a string, and appends it to the variable holding the entire contents of what will become the SRT file.

Writing the SRT File

Lastly, the contents of the $srtinfo variable are written to file.

Viewing the Results

At this point, it really depends on what you want to do with the SRT file and accompanying media one. Media players like VLC allow you to manually add an SRT file to a playing video, and most current televisions with USB will be happy to display the subtitles provided the filenames (without extension) match.
If you really want to go full (non-pron) hardcore, you could add the SRT info as a stream directly into the media file, using a tool like FFMPEG, which allows you to multiplex, or even to “burn” the subtitles onto the video. Using this method, vloggers really wanting to reach their audience could make multiple language subtitles available in this video file.

Summary

The combination of AWS services S3 and Transcribe, coupled with PowerShell and their AWS module for it, make it a relatively straightforward process to obtain a transcription of a media file which can be converted to SRT format for later use.

Also, as a final word, bear in mind that as a service meant for transcribing relatively silent environments with spoken (not sung) words, sometimes the results of a media file can be a little bit…misunderstood. Often with humerous results… 🙂

Marvin’s on his own

Let’s help him with some words

Thats Marvin SRTd

Wrong words but’s let’s karaoke anyway

Thanks for reading, and feedback always welcome.

Coming soon…a slight departure from the norm with an advanced version of this but using Golang.

Think i need to see about getting me another domain name…

Share

When Marvin Gaye Met Amazon Transcribe & PowerShell – Automating Subtitle Creation – Part II

“I Heard it Through the Grape Van”

With the background set in the previous post for what we’ll be aiming to achieve, it’s time to move forward with getting things into gear.
Todays post covers how to upload the media file to s3, create the Transcribe job to process it, and finally download the results locally.

Quick Recap

This projects demonstrates the use of the AWS Transcribe service and PowerShell to create an SRT (subtitle) file from a media file.

Our project makes use of:

  • PowerShell Core
  • AWS PowerShell Core Cmdlets
  • AWS S3
  • AWS Transcribe Service
  • An MP4 video file

Prerequisites

Before going into the nitty gritty, you need to ensure all of the following are in place:

  • An existing AWS account. You can sign up for a Free Tier account here
  • Credentials and configuration are setup. See here for information on how to do this.
  • PowerShell Core is installed (available cross platform here)
  • The AWS PowerShell Net Core module is installed.
  • You’ve a ready-to-hand MP4 media file
  • You have cloned the repo for the project from either its source or your own fork

Sequence of Events to Transcribe the File

The order of events that need to happen is relatively straightforward:

  • Upload the file to S3
  • Create a Transcribe job
  • Wait for job to finish
  • Download the JSON file

Upload the file to S3

We’ll start out defining some variables and defaults to make things a bit easier, then the Write-S3Object cmdlet takes care of itself:

Create a Transcribe job

All Transcribe jobs have an associated job name associated with this. For this script, I’ve used the GUID class to create a unique one. We define this and the name of the results file that will be used when it’s downloaded from a completed job. Then the Start-TRSTranscriptionJob cmdlet is used to initiate the task. The $s3uri variable is used to tell Transcribe where to get the file it is to process.

Wait for job to finish

A basic loop is put in place which checks the status of the Transcribe job every five seconds. The loops continues until the job status changes from IN_PROGRESS, indicating either a failure or completion of it.

Download the JSON file

When a job has successfully executed, visible by its COMPLETED status, it stores the result in an s3 bucket of its own choice. The location is not in your own personal bucket, and has an expiry life. By querying the TranscriptFileUri property of the job status, we can get the location where it is stored. You’ve then got the choice of using the S3 Manager cmdlet for downloading the file, or alternatively (in this case), simply with Invoke-Webrequest.

Part III will cover the largest part of the process, converting the job results into the SRT file we’ll use with the original video.
Thanks for reading!

Share

When Marvin Gaye Met Amazon Transcribe & PowerShell – Automating Subtitle Creation – Part I

“I Heard it Through the Grape Van”

It’s been a while since my last blog, so I had to try and think of something a bit more eye-catching than the previous ones for the title. 🙂  That said, the title and heading are actually very accurate… #mysterious

These set of posts cover how one of the AWS services, Transcribe, can be used, in this case in combo with PowerShell, to create a subtitles file for any video, which can then be used for viewing. There’s quite a bit of content, so as mentioned, it’s being split across several posts.

Todays post provides a background to the main parts that will be used. These are two AWS services Amazon S3 and Amazon Transcribe, a subtitle file, and AWS Tools for PowerShell Core Edition, and a video of the legend himself, Marvin Gaye.

Amazon Transcribe/S3

Amongst the plethora of services AWS offer is Transcribe, or to be more precise, Amazon Transcribe. Part of AWS’s group of Machine Learning offerings, Transcribe’s role is fairly straightforward. Feed it a supported media file (FLAC, MP3, MP4 or WAV) from a bucket on S3 and it will process the file, endeavoring to provide as best as possible a transcription of it. Upon successful completion of a job, a JSON formatted file becomes available for download.

The file itself contains a summary of the conversion at its beginning:

Which is then followed by a breakdown of the job. This consists either of data about the next word identified (start and end time, the word, a ‘confidence’ rating from the service that it has correctly identified the word, and its classification…

…or if its found an appropriate place that would use punctuation.

Unlike the other formats supported, MPG4 can also (and usually does), consist of one of more additional streams than audio. Typically this will be video content, but it might also include additional streams for other audio (think different languages, or directors/producers comments for example) or subtitles.

Subtitle Files

At their core, subtitle files simply contain textual descriptions of the content of its accompanying video file. This is typically dialogue, but also other notifications, such as the type of music being played, or other intonations. Accompanying these are timespan indicators, which are used to match this information up with the video content.

The most common file format in use is the Subrip format, better recognised by its extension of SRT. These files are arranged in a manner similar to below:

Line by line respectively, these consist of :
  • The numeric counter identifying each sequential subtitle
  • Start and end time for the subtitle to be visible, separated by the marker you see.
  • The text itself, typically between one and two lines, and ideally restricted to a number of characters per line
  • A blank line indicating the end of this sequence.

Looking at the two different forms of text data in Transcribe and SRT format respectively, you’ll probably have already noticed that the former contains enough information that should allow, with a bit of transformation, the output to be in the latters.

AWS Tools for PowerShell Core Edition

PowerShell Core is Microsoft’s cross platform implementation of PowerShell and as such can pretty much run on any platform that has .NET Core installed on it. AWS provide a module for this platform, AWS Tools for PowerShell Core Edition. Consisting of, at present, 4136 cmdlets, it pretty much covers all of the broad spectrum of services available from the provider. Amongst these are the set of ones for the Transcribe service, ironically numbering only three.

Marvin Gaye

Needing no introduction whatsoever, the posts over the next day or so make use of an MP4 file of the legend singing I Heard it Through the Grapevine acapella. If you really feel the need to follow along exactly, then its fairly straightforward to find and download. It’s most definitely worth a listen in any case if you’ve not heard it already.

With all the background set, part II will kick in properly with getting setup for the script and the beginning of its implementation.

 

Share

Using PowerShell to get data for Microsoft Ignite

I was registered for Ignite yesterday (yipeeee!!), and decided to take a look at the session list. 

Navigation and search is a bit of a chore, so I set out to see if I could get the information I needed via PowerShell. If so, I was free to obtain whatever data wanted quickly.

Here’s what i came up with. After the script a couple of examples of post querying the data are given. Note that instead of querying the web services each time for data, I’ve just downloaded all the data, and query it locally. This isn’t really best practice, but (IMO) the low size of the dataset mitigates this to some extent.

Recommendations for improving or additions are more than welcome. 🙂 It will be posted to GitHub shortly.

 

JeffreyQuery

SpeakersQuery

Share

Using PowerShell to make Azure Automation Graphical Runbooks – Part 1

Introduction

Last week saw the release of the Azure Automation Graphical SDK for creating graphical Runbooks programmatically. On its own, it is a library, represented by a single file,  Orchestrator.GraphRunbook.Model.dll. It contains the .NET components required to allow creation of a graphical runbook object, serialize it, then save it to a .graphrunbook file, suitable for importing into Azure Automation.

The SDK is by default installed to C:\Program Files (x86)\Microsoft Azure Automation Graphical Authoring SDK.

The examples given in the documentation that accompanies the download from Microsoft’s web page for this are in C#. This series will show how these same graphical runbooks can be generated in PowerShell.

Overview

The next few articles walk through how we can make a runbook, using an existing graphical runbook generated in the Azure Portal, Get-DiceThrow, as a point of reference. It then shows how an identical one can be programmatically made using PowerShell and the SDK.

The Existing Runbook

In the screenshot below is the graphical runbook we mentioned, Get-DiceThrow.

0 - GraphRunbook Design

Get-DiceThrow accepts as input two integer parameters, simulating the roll of two dice. It then identifies if one is greater than the other, or if they are both of the same value. Then, based on that, a message is displayed, describing the result.

When we’re looking at designing a graphical runbook via the SDK, it’s a good idea to first take a look at the structure of our existing one to get an insight of what it’s composed of. This helps later when development of the script starts.

To do this, the classes and properties that represent what we can see in Get-DiceThrow will be described.

Runbook Elements

Runbook

A runbook, or more specifically a graphical runbook, is repesented by an instance of the the GraphRunbook class. This describes the runbook in its entirety, and consists of the properties Activities, Links, and Parameters. This last one refers to the runbook’s input an output parmeters (see below).

1 - GraphRunbook

Runbook Input and Output Parameters

Input and output parameters are defined by creating an one or more instances of a Parameter class and setting the Paramters property of the Runbook class.

Parameters

The attributes for the name of the parameter, the parameters type, and whether it is mandatory or not are represented in the Parameter class by the properties of Name, Type, and Optional.

2 - Input and Output

Name, Type, and Optional

Workflows

Workflow script activities are represented by the WorkflowScriptActivity class. From the image below (anti-clockwise from Label) , the associated properties are Name, Description, CheckPointAfter, and Process.

3 - WorkFlow Script

Links

Links between activities use the aptly named class, Link. Links can either have conditions attached to them (restricting the use of the activity at the end of the link to situations where the condition’s expression evaluates to true), or be simple unconditional ones.

The settings in the image below (top to bottom) are referred to by the properties LinkType, Description,and Condition. Condition is another class, which contains the properties ConditionMode, to specify if a condition should be applied, and Expression for the conditional expression.

6 - Link

The next article will focus on the activities themselves, and the classes which inherit from them. This is probably the part that most of the development time will be spent on, since there is a substantial amount of configuration required.

Thanks for reading,

Tim

 

Share

Using Drag and Drop in PowerShell GUIs

Drag and drop is a wonderful facility that most of us will use every day without even thinking about it. This article shows how it can be implemented in PowerShell GUI applications.

For this particular script, we’ll implement drag and drop functionality to allow us to drag a file from explorer to a textbox in we’ve created. When the drag and drop operation is complete, the textbox will then show the path to the file.

You can find a copy of the PowerShell Studio form and exported PowerShell code at the PowerShell.Amsterdam repository.

Here is what we our application will do once we have completed it:

To achieve our objective, we need to make use of two events, DragOver, and DragDrop.

DragOver occurs when the mouse is over the control on which we wish to ‘drop’ our object. We’ll typically just use this for changing our pointer to show a move or copy operation is in operation.

DragDrop occurs once the actual operation is finished. That is, the object has been dragged into the form and over the control, and the mouse button released.

To implement this in our application, carry out the following:

  • Create a form
  • Create a Textbox on the form
  • Set the following properties for the Textbox
    • Name : txtDragandDrop
    • Label : Contents dragged:
    • AllowDrop : true

1 - form design and properties

We now define the events that will be processed, and their handlers.

  • DragDrop : txtDragandDrop_DragDrop
  • DragOver : txtDragandDrop_DragOver

2 - form design and events

For the event handler code, use the following :

If you are developing this application in PowerShell Studio, you will need to first export the code, either to a standalone EXE, or .PS1 file.

  • Run the application from either of the above.
  • Drag a file of you choice from explorer to the textbox.

The textbox will be populated with the filepath of the file that has been dragged and dropped over it.

4 - running

Thanks for reading!

Tim

Share

Reporting SQL Server Version Information

Pulled together some scripts to make a cmdlet which returns version information from one or more SQL servers. Remember to set the SQLServer parameter value correctly. It will be different depending on whether you are using a named instance. You can find a copy of the source code at my powershell.amsterdam gitlab repository.

I’ve used Invoke-Command in the script so that the function can also be run remotely against other systems.

With this complete, we can use the cmdlet, and expect output similar to the format below.

SQL - VersionInformation

Share

Asynchronously Save Images in the Clipboard to File

PowerShell v5.0 introduces us with two cmdlets, Get-Clipboard and Set-Clipboard which, just as their names suggest, allow us to obtain and set the contents of the clipboard. What’s not immediately apparent though is that these cmdlets can process more than just text in the clipboard.

In this blog, we’ll focus on using Get-Clipboard, and create a script which is used in combination with Register-EvengineEvent to save an image to file any time that one is detected in the clipboard during the current session. A repo containing the script is available at GitHub

Get-Clipboard supports the processing of several types of data that is in the clipboard, one of which is that of an image type. Examples of these would be print screens or copying an image created in Paint into the clipboard.

When an image is in the clipboard, and we use Get-Clipboard -Format Image, an object of Bitmap type (inherited from System.Drawing.Image) is returned. One of the methods this class provides is Save. Save has several overloads, one of which writes to a file and a variety of formats of your choice.

The script below requires use of the ISE, and uses Register-EngineEvent to register a scriptblock, which runs when a PowerShell.OnIdle event is raised. In the scriptblock, we check to see if there is any content in the clipboard of image type. If there is, we take the details of the current file open in ISE the editor, and use it to generate a unique filename. Then, the content of the clipboard is saved to this filename using the method mentioned above. Finally, we clear the clipboard to ensure that there we don’t end up in a continuous looping operation.

Thanks for reading!

Tim

Share