1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
|
# go-getter
[][travis]
[][appveyor]
[][godocs]
[travis]: http://travis-ci.org/hashicorp/go-getter
[godocs]: http://godoc.org/github.com/hashicorp/go-getter
[appveyor]: https://ci.appveyor.com/project/hashicorp/go-getter/branch/master
go-getter is a library for Go (golang) for downloading files or directories
from various sources using a URL as the primary form of input.
The power of this library is being flexible in being able to download
from a number of different sources (file paths, Git, HTTP, Mercurial, etc.)
using a single string as input. This removes the burden of knowing how to
download from a variety of sources from the implementer.
The concept of a _detector_ automatically turns invalid URLs into proper
URLs. For example: "github.com/hashicorp/go-getter" would turn into a
Git URL. Or "./foo" would turn into a file URL. These are extensible.
This library is used by [Terraform](https://terraform.io) for
downloading modules, [Otto](https://ottoproject.io) for dependencies and
Appfile imports, and [Nomad](https://nomadproject.io) for downloading
binaries.
## Installation and Usage
Package documentation can be found on
[GoDoc](http://godoc.org/github.com/hashicorp/go-getter).
Installation can be done with a normal `go get`:
```
$ go get github.com/hashicorp/go-getter
```
go-getter also has a command you can use to test URL strings:
```
$ go install github.com/hashicorp/go-getter/cmd/go-getter
...
$ go-getter github.com/foo/bar ./foo
...
```
The command is useful for verifying URL structures.
## URL Format
go-getter uses a single string URL as input to download from a variety of
protocols. go-getter has various "tricks" with this URL to do certain things.
This section documents the URL format.
### Supported Protocols and Detectors
**Protocols** are used to download files/directories using a specific
mechanism. Example protocols are Git and HTTP.
**Detectors** are used to transform a valid or invalid URL into another
URL if it matches a certain pattern. Example: "github.com/user/repo" is
automatically transformed into a fully valid Git URL. This allows go-getter
to be very user friendly.
go-getter out of the box supports the following protocols. Additional protocols
can be augmented at runtime by implementing the `Getter` interface.
* Local files
* Git
* Mercurial
* HTTP
* Amazon S3
In addition to the above protocols, go-getter has what are called "detectors."
These take a URL and attempt to automatically choose the best protocol for
it, which might involve even changing the protocol. The following detection
is built-in by default:
* File paths such as "./foo" are automatically changed to absolute
file URLs.
* GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically
changed to Git protocol over HTTP.
* BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically
changed to a Git or mercurial protocol using the BitBucket API.
### Forced Protocol
In some cases, the protocol to use is ambiguous depending on the source
URL. For example, "http://github.com/mitchellh/vagrant.git" could reference
an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this
URL.
Forced protocol can be done by prefixing the URL with the protocol followed
by double colons. For example: `git::http://github.com/mitchellh/vagrant.git`
would download the given HTTP URL using the Git protocol.
Forced protocols will also override any detectors.
In the absense of a forced protocol, detectors may be run on the URL, transforming
the protocol anyways. The above example would've used the Git protocol either
way since the Git detector would've detected it was a GitHub URL.
### Protocol-Specific Options
Each protocol can support protocol-specific options to configure that
protocol. For example, the `git` protocol supports specifying a `ref`
query parameter that tells it what ref to checkout for that Git
repository.
The options are specified as query parameters on the URL (or URL-like string)
given to go-getter. Using the Git example above, the URL below is a valid
input to go-getter:
github.com/hashicorp/go-getter?ref=abcd1234
The protocol-specific options are documented below the URL format
section. But because they are part of the URL, we point it out here so
you know they exist.
### Checksumming
For file downloads of any protocol, go-getter can automatically verify
a checksum for you. Note that checksumming only works for downloading files,
not directories, but checksumming will work for any protocol.
To checksum a file, append a `checksum` query parameter to the URL.
The paramter value should be in the format of `type:value`, where
type is "md5", "sha1", "sha256", or "sha512". The "value" should be
the actual checksum value. go-getter will parse out this query parameter
automatically and use it to verify the checksum. An example URL
is shown below:
```
./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21
```
The checksum query parameter is never sent to the backend protocol
implementation. It is used at a higher level by go-getter itself.
### Unarchiving
go-getter will automatically unarchive files into a file or directory
based on the extension of the file being requested (over any protocol).
This works for both file and directory downloads.
go-getter looks for an `archive` query parameter to specify the format of
the archive. If this isn't specified, go-getter will use the extension of
the path to see if it appears archived. Unarchiving can be explicitly
disabled by setting the `archive` query parameter to `false`.
The following archive formats are supported:
* `tar.gz` and `tgz`
* `tar.bz2` and `tbz2`
* `zip`
* `gz`
* `bz2`
For example, an example URL is shown below:
```
./foo.zip
```
This will automatically be inferred to be a ZIP file and will be extracted.
You can also be explicit about the archive type:
```
./some/other/path?archive=zip
```
And finally, you can disable archiving completely:
```
./some/path?archive=false
```
You can combine unarchiving with the other features of go-getter such
as checksumming. The special `archive` query parameter will be removed
from the URL before going to the final protocol downloader.
## Protocol-Specific Options
This section documents the protocol-specific options that can be specified
for go-getter. These options should be appended to the input as normal query
parameters. Depending on the usage of go-getter, applications may provide
alternate ways of inputting options. For example, [Nomad](https://www.nomadproject.io)
provides a nice options block for specifying options rather than in the URL.
## General (All Protocols)
The options below are available to all protocols:
* `archive` - The archive format to use to unarchive this file, or "" (empty
string) to disable unarchiving. For more details, see the complete section
on archive support above.
* `checksum` - Checksum to verify the downloaded file or archive. See
the entire section on checksumming above for format and more details.
### Local Files (`file`)
None
### Git (`git`)
* `ref` - The Git ref to checkout. This is a ref, so it can point to
a commit SHA, a branch name, etc. If it is a named ref such as a branch
name, go-getter will update it to the latest on each get.
* `sshkey` - An SSH private key to use during clones. The provided key must
be a base64-encoded string. For example, to generate a suitable `sshkey`
from a private key file on disk, you would run `base64 -w0 <file>`.
**Note**: Git 2.3+ is required to use this feature.
### Mercurial (`hg`)
* `rev` - The Mercurial revision to checkout.
### HTTP (`http`)
None
### S3 (`s3`)
S3 takes various access configurations in the URL. Note that it will also
read these from standard AWS environment variables if they're set. If
the query parameters are present, these take priority.
* `aws_access_key_id` - AWS access key.
* `aws_access_key_secret` - AWS access key secret.
* `aws_access_token` - AWS access token if this is being used.
#### Using IAM Instance Profiles with S3
If you use go-getter and want to use an EC2 IAM Instance Profile to avoid
using credentials, then just omit these and the profile, if available will
be used automatically.
#### S3 Bucket Examples
S3 has several addressing schemes used to reference your bucket. These are
listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro
Some examples for these addressing schemes:
- s3::https://s3.amazonaws.com/bucket/foo
- s3::https://s3-eu-west-1.amazonaws.com/bucket/foo
- bucket.s3.amazonaws.com/foo
- bucket.s3-eu-west-1.amazonaws.com/foo/bar
|