August 6, 2014

I have a new site!

I have a new blog for my code & programming posts, as well as links to projects I'm working on.  It's at http://npf.io.  This will remain my blog for personal posts, but all further coding related posts will be to npf.io.

See you there!

July 9, 2014

CI for Windows Go Packages with AppVeyor

I recently needed to update my npipe package, and since I want it to be production quality, that means setting up CI, so that people using my package can know it's passing tests.  Normally I'd use Travis CI or Drone.io for that, but npipe is a Windows-only Go package, and neither of the aforementioned services support running tests on Windows.

With some googling, I saw that Nathan Youngman had worked with AppVeyor to add Go support to their CI system.  The example on the blog talks about making a build.cmd file in your repo to enable Go builds, but I found that you can easily set up a Go build without having to put CI-specific files in your repo.

To get started with AppVeyor, just log into their site and tell it where to get your code (I logged in with Github, and it was easy to specify what repo of mine to test).  Once you choose the repo, go to the Settings page on AppVeyor for that repo.  Under the Environment tab on the left, set the clone directory to C:\GOPATH\src\<your import path> and set an environment variable called GOPATH to C:\GOPATH.  Under the build tab, set the build type to "SCRIPT" and the script type to "CMD", and make the contents of the script go get -v -d -t <your import path>/... (this will download the dependencies for your package).  In the test tab, set the test type to "SCRIPT", the script type to "CMD" and the script contents to go test -v -cover ./...  (this will run all the tests in verbose mode and also output the test coverage).

That's pretty much it.  AppVeyor will automatically run a build on commits, like you'd expect.  You can watch the progress on a console output on their page, and get a pretty little badge from the badges page.  It's free for open source projects, and seems relatively responsive from my admittedly limited experience.

This is a great boon for Go developers, so you can be sure your code builds and passes tests on Windows, with very little work to set it up.  I'm probably going to add this to all my production repos, even the ones that aren't Windows-only, to ensure my code works well on Windows as well as Linux.


July 7, 2014

Intro to BoltDB: Painless Performant Persistence

BoltDB is a pure Go persistence solution that saves data to a memory mapped file. I call it a persistence solution and not a database, because the word database has a lot of baggage associated with it that doesn't apply to bolt. And that lack of baggage is what makes bolt so awesome.

Bolt is just a Go package. There's nothing you need to install on the system, no configuration to figure out before you can start coding, nothing. You just go get github.com/boltdb/bolt and then import "github.com/boltdb/bolt".

All you need to fully use bolt as storage is a file name. This is fantastic from both a developer's point of view, and a user's point of view. I don't know about you, but I've spent months of work time over my career configuring and setting up databases and debugging configuration problems, users and permissions and all the other crap you get from more traditional databases like Postgres and Mongo. There's none of that with bolt. No users, no setup, just a file name. This is also a boon for users of your application, because *they* don't have to futz with all that crap either.

Bolt is not a relational database. It's not even a document store, though you can sort of use it that way. It's really just a key/value store... but don't worry if you don't really know what that means or how you'd use that for storage. It's super simple and it's incredibly flexible. Let's take a look.

Storage in bolt is divided into buckets. A bucket is simply a named collection of key/value pairs, just like Go's map. The name of the bucket, the keys, and the values are all of type []byte. Buckets can contain other buckets, also keyed by a []byte name.

... that's it. No, really, that's it. Bolt is basically a bunch of nested maps. And this simplicity is what makes it so easy to use. There's no tables to set up, no schemas, no complex querying language to struggle with. Let's look at a bolt hello world:

package main

import (
    "fmt"
    "log"

    "github.com/boltdb/bolt"
)

var world = []byte("world")

func main() {
    db, err := bolt.Open("/home/nate/foo/bolt.db", 0644, nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    key := []byte("hello")
    value := []byte("Hello World!")

    // store some data
    err = db.Update(func(tx *bolt.Tx) error {
        bucket, err := tx.CreateBucketIfNotExists(world)
        if err != nil {
            return err
        }

        err = bucket.Put(key, value)
        if err != nil {
            return err
        }
        return nil
    })

    if err != nil {
        log.Fatal(err)
    }

    // retrieve the data
    err = db.View(func(tx *bolt.Tx) error {
        bucket := tx.Bucket(world)
        if bucket == nil {
            return fmt.Errorf("Bucket %q not found!", world)
        }

        val := bucket.Get(key)
        fmt.Println(string(val))

        return nil
    })

    if err != nil {
        log.Fatal(err)
    }
}

// output:
// Hello World!
I know what you're thinking - that seems kinda long. But keep in mind, I fully handled all errors in at least a semi-proper way, and we're doing all this:

1.) creating a database
2.) creating some structure (the "world" bucket)
3.) storing data to the structure
4.) retrieving data from the structure.

I think that's not too bad in 54 lines of code.

So let's look at what that example is really doing. First we call bolt.Open to get the database. This will create the file if necessary, or open it if it exists.

All reads from or writes to the bolt database must be done within a transaction. You can have as many Readers in read-only transactions at the same time as you want, but only one Writer in a writable transaction at a time (readers maintain a consistent view of the DB while writers are writing).

To begin, we call db.Update, which takes a function to which it'll pass a bolt.Tx - bolt's transaction object. We then create a Bucket (since all data in bolt lives in buckets), and add our key/value pair to it. After the write transaction finishes, we start a read- only transaction with DB.View, and get the values back out.

What's great about bolt's transaction mechanism is that it's super simple - the scope of the function is the scope of the transaction. If the function passed to Update returns nil, all updates from the transaction are atomically stored to the database. If the function passed to Update returns an error, the transaction is rolled back. This makes bolt's transactions completely intuitive from a Go developer's point of view. You just exit early out of your function by returning an error as usual, and bolt Does The Right Thing. No need to worry about manually rolling back updates or anything, just return an error.

The only other basic thing you may need is to iterate over key/value pairs in a Bucket, in which case, you just call bucket.Cursor(), which returns a Cursor value, which has functions like Next(), Prev() etc that return a key/value pair and work like you'd expect.

There's a lot more to the bolt API, but most of the rest of it is more about database statistics and some stuff for more advanced usage scenarios... but the above is all you really need to know to start storing data in a bolt database.

For a more complex application, just storing strings in the database may not be sufficient, but that's ok, Go has your back there, too. You can easily use encoding/json or encoding/gob to serialize structs into the database, keyed by a unique name or id. This is what makes it easy for bolt to go from a key/value store to a document store - just have one bucket per document type. Again, the benefit of bolt is low barrier of entry. You don't have to figure out a whole database schema or install anything to be able to just start dumping data to disk in a performant and manageable way.

The main drawback of bolt is that there are no queries. You can't say "give me all foo objects with a name that starts with bar". You could make your own index in the database and keep it up to date manually. This could be as easy as a slice of IDs serialized into an "indices" bucket for a particular query. Obviously, this is where you start getting into the realm of developing your own relational database, but if you don't go overboard, it can be nice that all this code is just that - code. It's not queries in some external DSL, it's just code like you'd write for an in-memory data store.

Bolt is not for every application. You must understand your application's needs and if bolt's key/value style will be sufficient to fulfill those needs. If it is, I think you'll be very happy to use such a simple data store with so little mental overhead.

[edited to clarify reader/writer relationship] Bonus Gob vs. Json benchmark for storing structs in Bolt:
BenchmarkGobEncode  1000000       2191 ns/op
BenchmarkJsonEncode   500000       4738 ns/op
BenchmarkGobDecode  1000000       2019 ns/op
BenchmarkJsonDecode   200000      12993 ns/op
Code: http://play.golang.org/p/IvfDUGBpJ6

June 17, 2014

Autogenerate docs with this one dumb trick

Yesterday, I was trying to think of a way of automating some doc generation for my go packages. The specific task I wanted to automate was updating a badge in my package's README to show the test coverage. What I wanted was a way to run go test -cover, parse the results, and put the result in the correct spot of my README. My first thought was to write an application that would do that for me ... but then I'd have to run that instead of go test. What I realized I wanted was something that was "compatible with go test" - i.e. I want to run go test and not have to remember to run some special other command.

And that's when it hit me: What is a test in Go? A test is a Go function that gets run when you run "go test".  Nothing says your test has to actually test anything.  And nothing prevents your test from doing something permanent on your machine (in fact we usually have to bend over backwards to make sure our tests don't do anything permanent.  You can just write a test function that updates the docs for you.

I actually quite like this technique.  I often have some manual tasks after updating my code - usually updating the docs in the README with changes to the API, or changing the docs to show new CLI flags, etc.  And there's one thing I always do after I update my code - and that's run "go test".  If that also updates my docs, all the better.

This is how covergen was born.  https://github.com/natefinch/covergen

Covergen is a particularly heinous example of a test that updates your docs.  The heinous part is that it actually doubles the time it takes to run your tests... this is because that one test re-runs all the tests with -cover to get the coverage percent.  I'm not sure I'd actually release real code that used such a thing - doubling the time it takes to run your tests just to save a few seconds of copy and paste is pretty terrible.

However, it's a valid example of what you can do when you throw away testing convention and decide you want to write some code in a test that doesn't actually test anything, and instead just runs some automated tasks that you want run whenever anyone runs go test.  Just make sure the result is idempotent so you're not continually causing things to look modified to version control.

May 14, 2014

Diffing Go with Beyond Compare

I love Beyond Compare, it's an awesome visual diff/merge tool.  It's not free, but I don't care, because it's awesome.  However, there's no built-in configuration for Go code, so I made one.  Not sure what the venn diagram of Beyond Compare users and Go users looks like, it might be that I'm the one point of crossover, but just in case I'm not, here's the configuration file for Beyond Compare 3 for the Go programming language: http://play.golang.org/p/G6NWE0z1GC  (please forgive the abuse of the Go playground)

Just copy the text into a file and in Beyond Compare, go to Tools->Import Settings... and choose the file.  Please let me know if you have any troubles or suggested improvements.

May 13, 2014

Intro++ to Go Interfaces

Standard Interface Intro

Go's interfaces are one of it's best features, but they're also one of the most confusing for newbies.  This post will try to give you the understanding you need to use Go's interfaces and not get frustrated when things don't work the way you expect.  It's a little long, but a bunch of that is just code examples.

Go's interfaces are different than interfaces in other languages, they are implicitly fulfilled.  This means that you never need to mark your type as explicitly implementing the interface (like class CFoo implements IFoo).  Instead, your type just needs to have the methods defined in the interface, and the compiler does the rest.

For example:

type Walker interface {
    Walk(miles int)
}

type Camel struct {
    Name string
}

func (c Camel) Walk(miles int) {
     fmt.Printf("%s is walking %v miles\n", c.Name, miles)
}

func LongWalk(w Walker) {
     w.Walk(500)
     w.Walk(500)
}

func main() {
    c := Camel{"Bill"}
    LongWalk(c)
}

// prints
// Bill is walking 500 miles.
// Bill is walking 500 miles.

http://play.golang.org/p/erodX-JplO

Camel implements the Walker interface, because it has a method named Walk that takes an int and doesn't return anything.  This means you can pass it into the LongWalk function, even though you never specified that your Camel is a Walker.  In fact, Camel and Walker can live in totally different packages and never know about one another, and this will still work if a third package decides to make a Camel and pass it into LongWalk.

Non-Standard Continuation

This is where most tutorials stop, and where most questions and problems begin.  The problem is that you still don't know how the interfaces actually *work*, and since it's not actually that complicated, let's talk about that.

What actually happens when you pass Camel into LongWalk?

So, first off, you're not passing Camel into LongWalk.  You're actually assigning c, a value of type Camel to a value w of type Walker, and w is what you operate on in LongWalk.

Under the covers, the Walker interface (like all interfaces), would look more or less like this if it were in Go (the actual code is in C, so this is just a really rough approximation that is easier to read).

type Walker struct {
    type *InterfaceType
    data *void
}

type InterfaceType struct {
    valtype *gotype
    func0 *func
    func1 *func
    ...
}

All interfaces values are just two pointers - one pointer to information about the interface type, and one pointer to the data from the value you passed into the interface (a void* in C-like languages... this should probably be Go's unsafe.Pointer, but I liked the explicitness of two actual *'s in the struct to show it's just two pointers).

The InterfaceType contains a pointer to information about the type of the value that you passed into the interface (valtype).  It also contains pointers to the methods that are available on the interface. 

When you assign c to w, the compiler generates instructions that looks more or less like this (it's not actually generating Go, this is just an easier-to-read approximation):

data := c
w := Walker{ 
    type: &InterfaceType{ 
              valtype: &typeof(c), 
              func0: &Camel.Walk 
          }
    data: &data
}

When you assign your Camel value c to the Walker value w, the Camel type is copied into the interface value's Type.valtype field.  The actual data in the value of c is copied into a new place in memory, and w's Data field points at that memory location.

Implications of the Implementation

Now, let's look at the implications of this code.  First, interface values are very small - just two pointers.  When you assign a value to an interface, that value gets copied once, into the interface, but after that, it's held in a pointer, so it doesn't get copied again if you pass the interface around.

So now you know why you don't need to pass around pointers to interfaces - they're small anyway, so you don't have to worry about copying the memory, plus they hold your data in a pointer, so changes to the data will travel with the interface.

Interfaces Are Types

Let's look at Walker again, this is important:

type Walker interface

Note that first word there: type.  Interfaces are types, just like string is a type or Camel is a type.  They aren't aliases, they're not magic hand-waving, they're real types and real values which are distinct from the type and value that gets assigned to them.

Now, let's assume you have this function:

func LongWalkAll(walkers []Walker) {
    for _, w := range walkers {
        LongWalk(w)
    }
}

And let's say you have a caravan of Camels that you want to send on a long walk:

caravan := []Camel{ Camel{"Bill"}, Camel{"Bob"}, Camel{"Steve"}}

You want to pass caravan into LongWalkAll, will the compiler let you?  Nope.  Why is that?  Well, []Walker is a specific type, it's a slice of values of type Walker.  It's not shorthand for "a slice of anything that matches the Walker interface".  It's an actual distinct type, the way []string is different from []int.  The Go compiler will output code to assign a single value of Camel to a single value of Walker.  That's the only place it'll help you out.  So, with slices, you have to do it yourself:

walkers := make([]Walker, len(caravan))
for n, c := range caravan {
    walkers[n] = c
}
LongWalkAll(walkers)

However, there's a better way if you know you'll just need the caravan for passing into LongWalkAll:

caravan := []Walker{ Camel{"Bill"}, Camel{"Bob"}, Camel{"Steve"}}
LongWalkAll(caravan)


Note that this goes for any type which includes an interface as part of its definition: there's no automatic conversion of your func(Camel) into func(Walker) or map[string]Camel into map[string]Walker.  Again, they're totally different types, they're not shorthand, and they're not aliases, and they're not just a pattern for the compiler to match.

Interfaces and the Pointers That Satisfy Them

What if Camel's Walk method had this signature instead?

func (c *Camel) Walk(miles int)

This line says that the type *Camel has a function called Walk.  This is important: *Camel is a type.  It's the "pointer to a Camel" type.  It's a distinct type from (non-pointer) Camel.  The part about it being a pointer is part of its type.  The Walk method is on the type *Camel.  The Walk method (in this new incarnation) is not on the type Camel. This becomes important when you try to assign it to an interface.

c := Camel{"Bill"}
LongWalk(c)

// compiler output:
cannot use c (type Camel) as type Walker in function argument:
 Camel does not implement Walker (Walk method has pointer receiver)

To pass a Camel into LongWalk now, you need to pass in a pointer to a Camel:

c := &Camel{"Bill"}
LongWalk(c)

or

c := Camel{"Bill"}
LongWalk(&c)

Note that this true even though you can still call Walk directly on Camel:

c := Camel{"Bill"}
c.Walk(500) // this works

The reason you can do that is that the Go compiler automatically converts this line to (&c).Walk(500) for you.  However, that doesn't work for passing the value into an interface.  The reason is that the value in an interface is in a hidden memory location, and so the compiler can't automatically get a pointer to that memory for you (in Go parlance, this is known as being "not addressable").

Nil Pointers and Nil Interfaces

The interaction between nil interfaces and nil pointers is where nearly everyone gets tripped up when they first start with Go.

Let's say we have our Camel type with the Walk method defined on *Camel as above, and we want to make a function that returns a Walker that is actually a Camel (note that you don't need a function to do this, you can just assign a *Camel to a Walker, but the function is a good illustrative example):

func MakeWalker() Walker {
    return &Camel{"Bill"}
}

w := MakeWalker()
if w != nil {
    w.Walk(500)  // we will hit this
}

This works fine.  But now, what if we do something a little different:

func MakeWalker(c *Camel) Walker {
    return c
}

var c *Camel
w := MakeWalker(c)
if w != nil {
    // we'll get in here, but why?
    w.Walk(500)
}

This code will also get inside the if statement (and then panic, which we'll talk about in a bit) because the returned Walker value is not nil.  How is that possible, if we returned a nil pointer?  Well, let's go look back to the instructions that get generated when we assign a value to an interface.  

data := c
w := Walker{ 
    type: &InterfaceType{ 
              valtype: &typeof(c), 
              func0: &Camel.Walk 
          }
    data: &data
}

In this case, c is a nil pointer. However, that's a perfectly valid value to assign to the Walker's Data value, so it works just fine.  What you return is a non-nil Walker value, that has a pointer to a nil *Camel as its data.  So, of course, if you check w == nil, the answer is false, w is not nil... but then inside the if statement, we try to call Camel's walk:

func (c *Camel) Walk(miles int) {
     fmt.Printf("%s is walking %v miles\n", c.Name, miles)
}

And when we try to do c.Name, Go automatically turns that into (*c).Name, and the code panics with a nil pointer dereference error.

Hopefully this makes sense, given our new understanding of how interfaces wrap values, but then how do you account for nil pointers?  Assume you want MakeWalker to return a nil interface if it gets passed a nil Camel.  You have to explicitly assign nil to the interface:


func MakeWalker(c *Camel) Walker {
    if c == nil {
        return nil
    }
    return c
}

var c *Camel
w := MakeWalker(c)
if w != nil {
    // Yay, we don't get here!
    w.Walk(500)
}

And now, finally, the code is doing what we expect.  When you pass in a nil *Camel, we return a nil interface.  Here's an alternate way to write the function:


func MakeWalker(c *Camel) Walker {
    var w Walker
    if c != nil {
        w = c
    }
    return w
}

This is slightly less optimal, but it shows the other way to get a nil interface, which is to use the zero value for the interface, which is nil.

Note that you can have a nil pointer value that satisfies an interface.  You just need to be careful not to dereference the pointer in your methods.  For example, if *Camel's Walk method looked like this:

func (c *Camel) Walk(miles int) {
    fmt.Printf("I'm walking %d miles!", miles)
}

Note that this method does not dereference c, and therefore you can call it even if c is nil:

var c *Camel
c.Walk(500)
// prints "I'm walking 500 miles!"


Outro

I hope this article helps you better understand how interfaces works, and helps you avoid some of the common pitfalls and misconceptions newbies have about how interfaces work.  If you want more information about the internals of interfaces and some of the optimizations that I didn't cover here, read Russ Cox's article on Go interfaces, I highly recommend it.

April 10, 2014

Mocking functions in Go

Functions in Go are first class citizens, that means you can have a variable that contains a function value, and call it like a regular function.
printf := fmt.Printf
printf("This will output %d line.\n", 1)
This ability can come in very handy for testing code that calls a function which is hard to properly test while testing the surrounding code.  In Juju, we occasionally use function variables to allow us to stub out a difficult function during tests, in order to more easily test the code that calls it.  Here's a simplified example:
// in install/mongodb.go
package install

func SetupMongodb(path string) error {
     // suppose the code in this method modifies files in root
     // directories, mucks with the environment, etc...
     // Actions you actively don't want to do during most tests.
}

// in startup/bootstrap.go
package startup

func Bootstrap() error {
    ...
    path := getPath()
    if err := install.SetupMongodb(path); err != nil {
       return err
    }
    ...
}
So, suppose you want to write a test for Bootstrap, but you know SetupMongodb won't work, because the tests don't run with root privileges (and you don't want to setup mongodb on the dev's machine anyway).  What can you do?  This is where mocking comes in.

We just make a little tweak to Bootstrap:
package startup

var setupMongo = install.SetupMongodb

func Bootstrap() error {
    ...
    path := getRootDirPath()
    if err := setupMongo(path); err != nil {
       return err
    }
    ...
}
Now if we want to test Bootstrap, we can mock out the setupMongo function thusly:
// in startup/bootstrap_test.go
package startup

type fakeSetup struct {
    path string
    err error
}

func (f *fakeSetup) setup(path string) error {
    f.path = path
    return f.err
}

TestBootstrap(t *testing.T) {
    f := &fakeSetup{ err: errors.New("Failed!") }
    // this mocks out the function that Bootstrap() calls
    setupMongo = f.setup
    err := Bootstrap()
    if err != f.err {
        t.Fail("Error from setupMongo not returned. Expected %v, got %v", f.err, err)
    }
    expPath := getPath()
    if f.path != expPath {
        t.Fail("Path not correctly passed into setupMongo. Expected %q, got %q", expPath, f.path)
    }

    // and then try again with f.err == nil, you get the idea
}
Now we have full control over what happens in the setupMongo function, we can record the parameters that are passed into it, what it returns, and test that Bootstrap is at least using the API of the function correctly.

Obviously, we need tests elsewhere for install.SetupMongodb to make sure it does the right thing, but those can be tests internal to the install package, which can use non-exported fields and functions to effectively test the logic that would be impossible from an external package (like the setup package). Using this mocking means that we don't have to worry about setting up an environment that allows us to test SetupMongodb when we really only want to test Bootstrap.  We can just stub out the function and test that Bootstrap does everything correctly, and trust that SetupMongodb works because it's tested in its own package.

April 1, 2014

Effective Godoc

I started to write a blog post about how to get the most out of godoc, with examples in a repo, and then realized I could just write the whole post as godoc on the repo, so that's what I did.  Feel free to send pull requests if there's anything you see that could be improved.

I actually learned quite a lot writing this article, by exploring all the nooks and crannies of Go's documentation generation.  Hopefully you'll learn something too.

Either view the documentation on godoc.org:

https://godoc.org/github.com/natefinch/godocgo

or view it locally using the godoc tool:

go get code.google.com/p/go.tools/cmd/godoc
go get github.com/natefinch/godocgo
godoc -http=:8080

Then open a browser to http://localhost:8080/pkg/github.com/natefinch/godocgo

Enjoy!

March 28, 2014

Unused Variables in Go

The Go compiler treats unused variables as a compilation error. This causes much annoyance to some newbie Gophers, especially those used to writing languages that aren't compiled, and want to be able to be fast and loose with their code while doing exploratory hacking.

The thing is, an unused variable is often a bug in your code, so pointing it out early can save you a lot of heartache.

Here's an example:

50 func Connect(name, port string) error {
51    hostport := ""

52    if port == "" {

53        hostport := makeHost(name)

54        logger.Infof("No port specified, connecting on port 8080.")

55    } else {

56        hostport := makeHostPort(name, port)

57        logger.Infof("Connecting on port %s.", port)

58    }

59    // ... use hostport down here

60 }

Where's the bug in the above?  Without the compiler error, you'd run the code and have to figure out why hostport was always an empty string.  Did we pass in empty strings by accident?  Is there a bug in makeHost and makeHostPort?

With the compiler error, it will say "53, hostport declared and not used" and "56, hostport declared and not used"

This makes it a lot more obvious what the problem is... inside the scope of the if statement, := declares new variables called hostport.  These hide the variable from the outer scope, thus, the outer hostport never gets modified, which is what gets used further on in the function.

50 func Connect(name, port string) error {

51    hostport := ""

52    if port == "" {

53        hostport = makeHost(name)

54        logger.Infof("No port specified, connecting on port 8080.")

55    } else {

56        hostport = makeHostPort(name, port)

57        logger.Infof("Connecting on port %s.", port)

58    }

59    // ... use hostport down here

60 }

The above is the corrected code. It took only a few seconds to fix, thanks to the unused variable error from the compiler.  If you'd been testing this by running it or even with unit tests... you'd probably end up spending a non-trivial amount of time trying to figure it out.  And this is just a very simple example.  This kind of problem can be a lot more elaborate and hard to find.

And that's why the unused variable declaration error is actually a good thing.  If a value is important enough to be assigned to a variable, it's probably a bug if you're not actually using that variable.

Bonus tip:

Note that if you don't care about the variable, you can just assign it to the empty identifier directly:
_, err := computeMyVar()
This is the normal way to avoid the compiler error in cases where a function returns more than you need.

If you really want to silence the unused variable error and not remove the variable for some reason, this is the way to do it:
v, err := computeMyVar()
_ = v  // this counts as using the variable
Just don't forget to clean it up before committing.

All of the above also goes for unused packages.  And a similar tip for silencing that error:
_ = fmt.Printf // this counts as using the package

March 21, 2014

Go and Github

Francesc Campoy recently posted about how to work on someone else's Go repo from github.  His description was correct, but I think there's an easier way, and also one that might be slightly less confusing.

Let's say you want to work on your own branch of github.com/natefinch/gocog - here's the easiest way to do it:

  1. Fork github.com/natefinch/gocog on github
  2. mkdir -p $GOPATH/src/github.com/natefinch/gocog
  3. cd $GOPATH/src/github.com/natefinch/gocog
  4. git clone https://github.com/YOURNAME/gocog .
  5. (optional) go get github.com/natefinch/gocog

That's it.  Now you can work on the code, push/pull etc from your github repo as normal, and submit a pull request when you're done.

go get is useful for getting code that you want to use, but it's not very useful for getting code that you want to work on.  It doesn't set up source control.  git clone does.  What go get is handy for is getting the dependencies of a project, which is what step 5 does (only needed if the project relies on outside repos you don't already have).  (thanks to a post on G+ for reminding me that git clone won't get the dependencies)

Also note, the path on disk is the same as the original repo's URL, not your branch's URL.  That's intentional, and it's the key to making this work.  go get is the only thing that actually cares if the repo URL is the same as the path on disk.  Once the code is on disk, go build etc just expects import paths to be directories under $GOPATH.  The code expects to be under $GOPATH/src/github.com/natefinch/gocog because that's what the import statements say it should be.  There's no need to change import paths or anything wacky like that (though it does mean that you can't have both the original version of the code and your branch coexisting in the same $GOPATH).

Note that this is actually the same procedure that you'd use to work on your own code from github, you just change step 1 to "create the repo in github".  I prefer making the repo in github first because it lets me set up the license, the readme, and the .gitignore with just a few checkboxes, though obviously that's optional if you want to hack locally first.  In that case, just make sure to set up the path under gopath where it would go if you used go get, so that go get will work correctly when you decide to push up to github.

(updated to mention using go get after git clone)