Beginnings in Golang and AWS – Part II – Uploading to S3

Introduction

With the preambling and prerequisities of Part I out of the way, we can now begin working on writing some code to allow us to upload the MP4 file to an S3 bucket.

In this post, we’ll cover the format of a Go package, how to add packages to an installation of Go, the import statement, and lastly how we go about parsing command line options.

It might not seem a great deal of code, but there’s quite a lot of concepts covered here that are essential to understanding how Go works. We’ll then be primed for the final part of the series on S3, which will cover the rest of the code, compiling, running, and using this program.

Uploading a File to S3

I’ll show the complete code first, and then break it down into parts.

Code Breakdown

Let’s go through the code and get a feel for what’s happen here.

VERY important! Go is case sensitive. Capitalization is treated differently, both from a name interpretation, and also operational point of view.

The Package Declaration

Every Go program consists of one or more packages. For a program to run (as opposed to being a resource for another program), it requires a main package.

Define Packages to be Used

Multiple packages exist for Go, both as part of a default installation, and also from the community.

The import statement tells Go what packages (and consequently resources such as functions and types) are available to the program. We can either do separate import statements, or group them together like above.

Go has a default package directory setting for packages not included in the default installation, from which it attempts to find the package (typically ~/go/src).

For example, github.com/aws/aws-sdk-go/aws, referred to in the import statement above, is located at the following location under my home directory:

When you want to use a resource in a package, such as a function or type, you need to refer to it including the package name. So if we wanted to use the Printf function within the fmt package with write a message, an example of this would be:

fmt.Printf(“No Hello World today”)

Define the Main Function

The entry point for a file to be executable (as opposed to solely a resource) is the main() function. The code executed within the function, represented by the dots, is enclosed within { and } braces.

Configure Command Line Parameter Parsing

When we execute this program from the command line, we want to include parameters which will define both the s3 bucket we want to upload to, and also the source file. The values need to be parsed and assigned to variables. To make it easier, we also want to provide some help text for people running the program.

Several things happen with the above code, so let’s go through them.

  • Both the bucket and filename variables are defined. Go normally requires a variable and its type to be pre-declared before it can be used. However, it is possible to create a variable with its type, and assign a value to it by using  :=  Quite simply what this does is leave it to the right of the operator to provide and type and value information. In this case, it is using the result of the String function in the flag package.
  • We use the flag package. The flag package has functions that allow us to parse the command line. We use flag.String to define a string flag with specified name, default value, and usage string. The return value is a reference to the memory address (aka pointer) which stores the value of the flag.
  • The Parse function is called. This carries out the processing of the command line parameters. This function sets the values at the memory location referred to by bucket and filename
  • It’s worthwhile mentioning that the output that will be generated if help is requested on our program, once compiled, is defined in this code as well. We’ll see in the last part on the S3 Uploader just exactly how this works.
  • You also might be wondering why the function name is capitalized. This is because in order for a resource in a package to be used by another, the initial letter must be a capital one. This marks it as “exportable”, allowing its use elsewhere.

Conclusion

In this post, we’ve covered a lot of topics, such as how we can use existing packages with our go program, how packages are stored locally, the effect that using lower and uppercase letters can have, the requirements for a program in go, and the import statement. We also began to delve in assignment by inference, pointers, flags  and how we can parse them.

Wit these out of the way, we’re primed and pretty much tickety boo for the final part of the series on S3, which will cover the rest of the code, introducing further concepts and syntax, including compiling, running, and using this program.

Thanks for reading!

Share

Beginnings in Golang and AWS – Part I

Part I – Introduction, Go, and Prerequisites

The series of blogs describe my journey learning Golang so far, with particular reference to AWS services. In them, I’ll be covering how I was able to achieve a task with AWS services with Go, but also what I learnt about Go at the same time. In addition to some services already mentioned in previous blogs, we’ll also be covering Lex, API Gateway, Lambda, and Alexa Skills.

Introduction

DSL’s aside, it’s been quite some time since I’ve endeavored to learn a new language in IT. Aside from a smattering of parsing through some Ruby and C# code on occasions and a bit of Python with BOTO3 for AWS stuff, it’d be fair to say the last one I learned to any level of depth was PowerShell.

Go

Already providing support for Node.js, Java, C# and Python with Lambda, AWS announced in January this year the addition of Go (short and more commonly used form of Golang). I’ve seen a lot of enthusiasm around the communities about Go, and with it marked as cross platform, fast, and of a relatively simple nature (sorta), I decided I’d give this a bash (no pun intended).

Whether it’s because I’m over-enthusiastic or (more likely) of a completely disorganized mind, my usual modus operandi involves skipping “hello world” learning completely and just diving in. Most certainly not a purist compliant approach…
I figured a good way to try this would be to adapt some of the PowerShell scripts I’ve written previously that feature here. Unoriginal for sure, but as it’s fresh in my mind, why not try with Golang to perform the creation of an SRT file?

Previously discussed in the PowerShell blogs, the task effectively comprises uploading a media file to an S3 bucket, creating a Transcribe job, obtaining the results, and then converting them.

As we’ll see later in this series of blogs though, additional options are available for how we carry some of these tasks out, making further automation and input sources possible.

For now though, we’ll get the pre-requisites out of the way.

Preparation

  • An existing AWS account. You can sign up for a Free Tier account here
  • Credentials and configuration are setup. See here for information on how to do this.
  • Go is installed (see below)
  • AWS SDK for Go is installed (see below)
  • You have cloned the repo for the project from either its source or your own fork
  • You’ve a ready-to-hand MP4 media file
  • You have a suitable development environment, such as VS Code

Install Go

Naturally, you are going to need Go installed. You can find your appropriate binary at the downloads page of golang.org. I’ll leave it to you to follow the specific installation instructions and any steps needed to fulfill requirements.

Install the AWS SDK for Go

You’ll also need the AWS SDK. Go has a really nice way to add packages to your environment, which is simply achieved by using the go get xxxxx, where xxxxx is the location of the package from the command line. Go handles the rest.

For installation of the AWS SDK for Go, simply use the following:

go get github.com/aws/aws-sdk-go/

Install VS Code (optional)

I currently use VS Code, which has some nice extensions to do tasks such as formatting, and automatic adding and removal of import commands. If you’re not currently using a development environment, or just simply fancy trying it out, you can obtain the binaries (available for Windows, OSX, and Linux) from here:

Add the Go Extention to VS Code

Follow the instructions and after installation is complete, launch VS Code. Add the extention ms-vscode.go, which provides features such as mentioned above within the development environment.

Conclusion

With the prerequisites now in place for us to use Go with AWS services, we can begin the process of putting together code which will help us achieve the tasks we’ve already mentioned.

In the next blog, we’ll dive into writing a script that will upload our media file to S3 using the SDK, and at the same time learning about a bit about Go.

Thanks for reading!

Share