二三事

emacs-go-tag 有两个依赖项， gomodifytags 和 go-mode.el，若使用emacs做Go开发 go-mode.el 应该已经在使用了。 gomodifytags安装如下:

go get -u github.com/fatih/gomodifytags

目前emacs-go-tag已发布到了MELPA上，可使用如下命令安装： M-x package-install [RET] go-tag [RET] 若是没发现go-tag,你需要使用如下命令来刷新,然后在使用上面的命令进行安装: M-x package-refresh-contents [RET]

目前 gomodifytags 支持三种转换如下：

• snakecase: BaseDomain -> base_domain
• camelcase: BaseDomain -> baseDomain
• lispcase: BaseDomain -> base-domain

默认使用 snakecase , 若你准备使用 camelcase, 可配置如下：

(setq go-tag-args (list "-transform" "camelcase"))

快捷键配置示例：

(with-eval-after-load 'go-mode
  (define-key go-mode-map (kbd "C-c t") #'go-tag-add)
  (define-key go-mode-map (kbd "C-c T") #'go-tag-remove))

若是你有使用 vim-go 的 GoAddTags 和 GoRemoveTags 命令的经验， 那么你会很快上手 emacs-go-tag, 因为两者的行为完全一致（若有不同，请反馈）。

感谢 GoCN每日新闻 的每日推送，让我得以读到 《Go 终极指南：编写一个 Go 工具》, 然后我才得以写出这个emacs-plugin。 这里给出项目地址：https://github.com/brantou/emacs-go-tag, 欢迎大家来使用和来提问题，若是你觉得对你有帮助， 就给颗星 吧。

go get -u github.com/josharian/impl

impl -h

Usage of impl:
  -dir string
      package source directory, useful for vendored code

impl 'f *File' io.ReadWriteCloser

1
2
3
4
5
6
7
8
9
10
11

func (f *File) Read(p []byte) (n int, err error) {
  panic("not implemented")
}

func (f *File) Write(p []byte) (n int, err error) {
  panic("not implemented")
}

func (f *File) Close() error {
  panic("not implemented")
}

请参见之前的文章gomodifytags。

go get -u github.com/lukehoban/go-outline

go-outline -h

Usage of go-outline:
  -f string
      the path to the file to outline
  -imports-only
      parse imports only
  -src string
      source code of the file to outline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

package main

import "fmt"

type Outline struct {
  Label string
  Type  string
}

func (ol *Outline) Label() string {
  return ol.Label
}

const (
  name = "brantou"
)

const email = "brantou89@gmail.com"

var (
  cmd  string
  args string
)

func main() {
  fmt.Printf("hello world!")
}

go-outline -f main.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

[
    {
	"label": "main",
	"type": "package",
	"start": 1,
	"end": 290,
	"children": [
	    {
		"label": "\"fmt\"",
		"type": "import",
		"start": 23,
		"end": 28
	    },
	    {
		"label": "Outline",
		"type": "type",
		"start": 35,
		"end": 83
	    },
	    {
		"label": "Label",
		"type": "function",
		"receiverType": "*Outline",
		"start": 85,
		"end": 140
	    },
	    {
		"label": "name",
		"type": "constant",
		"start": 152,
		"end": 156
	    },
	    {
		"label": "email",
		"type": "constant",
		"start": 178,
		"end": 183
	    },
	    {
		"label": "cmd",
		"type": "variable",
		"start": 217,
		"end": 220
	    },
	    {
		"label": "args",
		"type": "variable",
		"start": 231,
		"end": 235
	    },
	    {
		"label": "main",
		"type": "function",
		"start": 246,
		"end": 290
	    }
	]
    }
]

go-outline -f main.go -imports-only

go get -u github.com/newhook/go-symbols

go-symbols -h

Usage of go-symbols:
  -tags build tags
      a list of build tags to consider satisfied during the build. For more information about build tags, see the description of build constraints in the documentation for the go/build package

go-symbols $GOPATH/src/github.com/astaxie/beego beego

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

[
 {
  "name": "BeegoOutput",
  "kind": "type",
  "package": "context",
  "path": "/home/parallels/.gvm/pkgsets/go1.9/global/src/github.com/astaxie/beego/context/output.go",
  "line": 35,
  "character": 0
 },
 {
  "name": "BeegoInput",
  "kind": "type",
  "package": "context",
  "path": "/home/parallels/.gvm/pkgsets/go1.9/global/src/github.com/astaxie/beego/context/input.go",
  "line": 43,
  "character": 0
 },
 {
  "name": "NewBeegoRequest",
  "kind": "func",
  "package": "httplib",
  "path": "/home/parallels/.gvm/pkgsets/go1.9/global/src/github.com/astaxie/beego/httplib/httplib.go",
  "line": 80,
  "character": 0
 },
 {
  "name": "BeegoHTTPSettings",
  "kind": "type",
  "package": "httplib",
  "path": "/home/parallels/.gvm/pkgsets/go1.9/global/src/github.com/astaxie/beego/httplib/httplib.go",
  "line": 130,
  "character": 0
 },
 {
  "name": "BeegoHTTPRequest",
  "kind": "type",
  "package": "httplib",
  "path": "/home/parallels/.gvm/pkgsets/go1.9/global/src/github.com/astaxie/beego/httplib/httplib.go",
  "line": 146,
  "character": 0
 }
]

An autocompletion daemon for the Go programming language.

go get -u -v github.com/nsf/gocode

gocode -h

Usage: gocode [-s] [-f=<format>] [-in=<path>] [-sock=<type>] [-addr=<addr>]
     <command> [<args>]

Flags:
  -addr string
      address for tcp socket (default "127.0.0.1:37373")
  -debug
      enable server-side debug mode
  -f string
      output format (vim | emacs | nice | csv | csv-with-package | json) (default "nice")
  -in string
      use this file instead of stdin input
  -profile int
      port on which to expose profiling information for pprof; 0 to disable profiling
  -s	run a server instead of a client
  -sock string
      socket type (unix | tcp) (default "unix")

Commands:
  autocomplete [<path>] <offset>     main autocompletion command
  close                              close the gocode daemon
  status                             gocode daemon status report
  drop-cache                         drop gocode daemon's cache
  set [<name> [<value>]]             list or set config options

Print where symbols are defined in Go source code.

go get -u -v github.com/rogpeppe/godef

godef -h

usage: godef [flags] [expr]
  -A	print all type and members information
  -a	print public type and member information
  -acme
      use current acme window
  -debug
      debug mode
  -f string
      Go source filename
  -i	read file from stdin
  -json
      output location in JSON format (-t flag is ignored)
  -o int
      file offset of identifier in stdin (default -1)
  -t	print type information

Retrieve documentation for items in Go source code.

go get -u -v github.com/zmb3/gogetdoc

gogetdoc -h

Usage of gogetdoc
  -cpuprofile string
      write cpu profile to file
  -json
      enable extended JSON output
  -linelength int
      maximum length of a line in the output (in Unicode code points) (default 80)
  -modified
      read an archive of modified files from standard input
  -pos string
      Filename and byte offset of item to document, e.g. foo.go:#123
  -tags build tags
      a list of build tags to consider satisfied during the build. For more information about build tags, see the description of build constraints in the documentation for the go/build package
  -u	show unexported fields

The archive format for the -modified flag consists of the file name, followed
by a newline, the decimal file size, another newline, and the contents of the file.

This allows editors to supply gogetdoc with the contents of their unsaved buffers.

Golint is a linter for Go source code.

go get -u github.com/golang/lint/golint

golint -h

Usage of golint:
	golint [flags] # runs on package in current directory
	golint [flags] [packages]
	golint [flags] [directories] # where a '/...' suffix includes all sub-directories
	golint [flags] [files] # all must belong to a single package
Flags:
  -min_confidence float
      minimum confidence of a problem to print it (default 0.8)
  -set_exit_status
      set exit status to 1 if any issues are found

golint google.go

google.go:19:1: exported function Search should have comment or be unexported

Gopkgs is tools that provide list of available Go packages that can be imported. This are alternative to go list all, it just faster.

go get github.com/uudashr/gopkgs/cmd/gopkgs

gopkgs -help

Usage of gopkgs:
  -format string
      custom output format (default "{ {.ImportPath} }")
  -help
      show this message


Use -format to custom the output using template syntax. The struct being passed to template is:

  type Pkg struct {
      Dir        string // directory containing package sources
      ImportPath string // import path of package in dir
      Name       string // package name
  }

gopkgs -format "{ {.Name} };{ {.ImportPath} }"

go get -u -v golang.org/x/tools/cmd/gorename

gorename -help

gorename: precise type-safe renaming of identifiers in Go source code.

Usage:

 gorename (-from <spec> | -offset <file>:#<byte-offset>) -to <name> [-force]

You must specify the object (named entity) to rename using the -offset
or -from flag.  Exactly one must be specified.

Flags:

  -offset    specifies the filename and byte offset of an identifier to rename.
             This form is intended for use by text editors.

  -from      specifies the object to rename using a query notation;
             This form is intended for interactive use at the command line.
             A legal -from query has one of the following forms:

    "encoding/json".Decoder.Decode        method of package-level named type
    (*"encoding/json".Decoder).Decode     ditto, alternative syntax
    "encoding/json".Decoder.buf           field of package-level named struct type
    "encoding/json".HTMLEscape            package member (const, func, var, type)
    "encoding/json".Decoder.Decode::x     local object x within a method
    "encoding/json".HTMLEscape::x         local object x within a function
    "encoding/json"::x                    object x anywhere within a package
    json.go::x                            object x within file json.go

             Double-quotes must be escaped when writing a shell command.
             Quotes may be omitted for single-segment import paths such as "fmt".

             For methods, the parens and '*' on the receiver type are both
             optional.

             It is an error if one of the ::x queries matches multiple
             objects.

  -to        the new name.

  -force     causes the renaming to proceed even if conflicts were reported.
             The resulting program may be ill-formed, or experience a change
             in behaviour.

             WARNING: this flag may even cause the renaming tool to crash.
             (In due course this bug will be fixed by moving certain
             analyses into the type-checker.)

  -d         display diffs instead of rewriting files

  -v         enables verbose logging.

gorename automatically computes the set of packages that might be
affected.  For a local renaming, this is just the package specified by
-from or -offset, but for a potentially exported name, gorename scans
the workspace ($GOROOT and $GOPATH).

gorename rejects renamings of concrete methods that would change the
assignability relation between types and interfaces.  If the interface
change was intentional, initiate the renaming at the interface method.

gorename rejects any renaming that would create a conflict at the point
of declaration, or a reference conflict (ambiguity or shadowing), or
anything else that could cause the resulting program not to compile.


Examples:

$ gorename -offset file.go:#123 -to foo

  Rename the object whose identifier is at byte offset 123 within file file.go.

$ gorename -from '"bytes".Buffer.Len' -to Size

  Rename the "Len" method of the *bytes.Buffer type to "Size".

---- TODO ----

Correctness:
- handle dot imports correctly
- document limitations (reflection, 'implements' algorithm).
- sketch a proof of exhaustiveness.

Features:
- support running on packages specified as *.go files on the command line
- support running on programs containing errors (loader.Config.AllowErrors)
- allow users to specify a scope other than "global" (to avoid being
  stuck by neglected packages in $GOPATH that don't build).
- support renaming the package clause (no object)
- support renaming an import path (no ident or object)
  (requires filesystem + SCM updates).
- detect and reject edits to autogenerated files (cgo, protobufs)
  and optionally $GOROOT packages.
- report all conflicts, or at least all qualitatively distinct ones.
  Sometimes we stop to avoid redundancy, but
  it may give a disproportionate sense of safety in -force mode.
- support renaming all instances of a pattern, e.g.
  all receiver vars of a given type,
  all local variables of a given type,
  all PkgNames for a given package.
- emit JSON output for other editors and tools.

go get -u -v github.com/godoctor/godoctor

godoctor -h

Usage: godoctor [<flag> ...] <refactoring> [<args> ...]

Each <flag> must be one of the following:
    -complete Output entire modified source files instead of displaying a diff
    -doc      Output documentation (install, user, man, or vim) and exit
    -file     Filename containing an element to refactor (default: stdin)
    -json     Accept commands in OpenRefactory JSON protocol format
    -list     List all refactorings and exit
    -pos      Position of a syntax element to refactor (default: entire file)
    -scope    Package name(s), or source file containing a program entrypoint
    -v        Verbose: list affected files
    -vv       Very verbose: list individual edits (implies -v)
    -w        Modify source files on disk (write) instead of displaying a diff


The <refactoring> argument determines the refactoring to perform:
    rename          Changes the name of an identifier
    extract         Extracts statements to a new function/method
    var             Extracts an expression, assigning it to a variable
    toggle          Toggles between a var declaration and := statement
    godoc           Adds stub GoDoc comments where they are missing

The <args> following the refactoring name vary depending on the refactoring.

To display usage information for a particular refactoring, such as rename, use:
    %% godoctor rename

For complete usage information, see the user manual: http://gorefactor.org/doc.html

A gofmt/goimports-like tool for Go programmers that fills in Go return statements with zero values to match the func return types

go get -u sourcegraph.com/sqs/goreturns

1
2
3
4
5

package goreturnsdemo

import "errors"

func F() (int, error) { return errors.New("foo") }

goreturns -h

usage: goreturns [flags] [path ...]
  -b	remove bare returns
  -d	display diffs instead of rewriting files
  -e	report all errors (not just the first 10 on different lines)
  -i	run goimports on the file prior to processing (default true)
  -l	list files whose formatting differs from goreturns's
  -local string
put imports beginning with this string after 3rd-party packages (see goimports)
  -p	print non-fatal typechecking errors to stderr
  -w	write result to (source) file instead of stdout

goreturns goreturnsdemo.go

1
2
3
4
5

package goreturnsdemo

import "errors"

func F() (int, error) { return 0, errors.New("foo") }

goreturns -d goreturnsdemo.go

1
2
3
4
5
6
7
8
9
10
11

diff goreturnsdemo.go gofmt/goreturnsdemo.go
--- /tmp/gofmt948886532       2017-11-26 18:59:03.671955645 +0800
+++ /tmp/gofmt584282515       2017-11-26 18:59:03.671955645 +0800
@@ -1,6 +1,5 @@
 package goreturnsdemo

-
 import "errors"

-func F() (int, error) { return errors.New("foo") }
+func F() (int, error) { return 0, errors.New("foo") }

Generate Go tests from your source code.

go get -u -v github.com/cweill/gotests/...

gotests -h

Usage of gotests:
  -all
      generate tests for all functions and methods
  -excl string
      regexp. generate tests for functions and methods that don't match. Takes precedence over -only, -exported, and -all
  -exported
      generate tests for exported functions and methods. Takes precedence over -only and -all
  -i	print test inputs in error messages
  -nosubtests
      disable generating tests using the Go 1.7 subtests feature
  -only string
      regexp. generate tests for functions and methods that match only. Takes precedence over -all
  -w	write output to (test) files instead of stdout

gotests -all ./

guru: a tool for answering questions about Go source code.

godepgraph is a program for generating a dependency graph of Go packages.

go get github.com/kisielk/godepgraph

godepgraph -h

Usage of godepgraph:
  -d	show dependencies of packages in the Go standard library
  -horizontal
      lay out the dependency graph horizontally instead of vertically
  -i string
      a comma-separated list of packages to ignore
  -l int
      max level of go dependency graph (default 256)
  -o string
      a comma-separated list of prefixes to include
  -p string
      a comma-separated list of prefixes to ignore
  -s	ignore packages in the Go standard library
  -t	include test packages
  -tags string
      a comma-separated list of build tags to consider satisified during the build

godepgraph github.com/kisielk/godepgraph

digraph godep {
_0 [label="flag" style="filled" color="palegreen"];
_1 [label="fmt" style="filled" color="palegreen"];
_2 [label="github.com/kisielk/godepgraph" style="filled" color="paleturquoise"];
_2 -> _0;
_2 -> _1;
_2 -> _3;
_2 -> _4;
_2 -> _5;
_2 -> _6;
_2 -> _7;
_3 [label="go/build" style="filled" color="palegreen"];
_4 [label="log" style="filled" color="palegreen"];
_5 [label="os" style="filled" color="palegreen"];
_6 [label="sort" style="filled" color="palegreen"];
_7 [label="strings" style="filled" color="palegreen"];
}

godepgraph github.com/kisielk/godepgraph | dot -Tpng -o godepgraph.png

go get -u github.com/TrueFurby/go-callvis

go-callvis -h

Usage of go-callvis:
  -debug
      Enable verbose log.
  -focus string
      Focus package with name or import path. (default "main")
  -group string
      Grouping functions by [pkg, type] (separate multiple by comma).
  -ignore string
      Ignore package paths with prefix (separate multiple by comma).
  -limit string
      Limit package paths to prefix. (separate multiple by comma)
  -minlen uint
      Minimum edge length (for wider output). (default 2)
  -nodesep float
      Minimum space between two adjacent nodes in the same rank (for taller output). (default 0.35)
  -nostd
      Omit calls to/from std packages.
  -tests
      Include test code.
  -version
      Show version and exit.

go-callvis github.com/TrueFurby/go-callvis | dot -Tpng -o go-callvis.png

go get -u github.com/KyleBanks/depth/cmd/depth

depth -h

Usage of depth:
  -explain string
      If set, show which packages import the specified target
  -internal
      If set, resolves dependencies of internal (stdlib) packages.
  -json
      If set, outputs the depencies in JSON format.
  -max int
      Sets the maximum depth of dependencies to resolve.
  -test
      If set, resolves dependencies used for testing.

depth github.com/KyleBanks/depth/cmd/depth

github.com/KyleBanks/depth/cmd/depth
  ├ encoding/json
  ├ flag
  ├ fmt
  ├ io
  ├ os
  ├ strings
  └ github.com/KyleBanks/depth
    ├ bytes
    ├ errors
    ├ go/build
    ├ os
    ├ path
    ├ sort
    └ strings
12 dependencies (11 internal, 1 external, 0 testing).

depth -internal strings

strings
  ├ errors
  ├ internal/cpu
  ├ io
    ├ errors
    └ sync
      ├ internal/race
        └ unsafe
      ├ runtime
        ├ runtime/internal/atomic
          └ unsafe
        ├ runtime/internal/sys
        └ unsafe
      ├ sync/atomic
        └ unsafe
      └ unsafe
  ├ unicode
  └ unicode/utf8
12 dependencies (12 internal, 0 external, 0 testing).
strings
  ├ errors
  ├ internal/cpu
  ├ io
  ├ unicode
  └ unicode/utf8
5 dependencies (5 internal, 0 external, 0 testing).

go get -u -v mvdan.cc/interfacer

interfacer -h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

package single

func Empty() {
}

type Closer interface {
  Close()
}

type ReadCloser interface {
  Closer
  Read()
}

func Basic(c Closer) {
  c.Close()
}

func BasicWrong(rc ReadCloser) { // WARN rc can be Closer
  rc.Close()
}

func OtherWrong(s St) { // WARN s can be Closer
  s.Close()
}

interfacer simple.go

simple.go:24:19: undeclared name: St

go get -u github.com/stripe/safesql

safesql

Usage: safesql [-q] [-v] package1 [package2 ...]
  -q	Only print on failure
  -v	Verbose mode

go get github.com/opennota/check/cmd/aligncheck

aligncheck  github.com/astaxie/beego

github.com/astaxie/beego: config.go:33:6: struct Config could have size 520 (currently 552)
github.com/astaxie/beego: config.go:51:6: struct Listen could have size 120 (currently 152)
github.com/astaxie/beego: config.go:71:6: struct WebConfig could have size 264 (currently 288)
github.com/astaxie/beego: config.go:89:6: struct SessionConfig could have size 104 (currently 128)
github.com/astaxie/beego: controller.go:67:6: struct Controller could have size 224 (currently 232)
github.com/astaxie/beego: router.go:124:6: struct ControllerRegister could have size 168 (currently 176)

go get github.com/opennota/check/cmd/structcheck

structcheck -h

Usage of structcheck:
  -a	Count assignments only
  -e	Report exported fields
  -t	Load test files too
  -tags string
      Build tags

structcheck fmt

fmt: ~/.gvm/gos/go1.9/src/fmt/scan.go:169:2: fmt.ssave.nlIsEnd
fmt: ~/.gvm/gos/go1.9/src/fmt/scan.go:170:2: fmt.ssave.nlIsSpace
fmt: ~/.gvm/gos/go1.9/src/fmt/scan.go:171:2: fmt.ssave.argLimit
fmt: ~/.gvm/gos/go1.9/src/fmt/scan.go:172:2: fmt.ssave.limit
fmt: ~/.gvm/gos/go1.9/src/fmt/scan.go:173:2: fmt.ssave.maxWid
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:30:2: fmt.fmtFlags.zero
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:35:2: fmt.fmtFlags.plusV
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:36:2: fmt.fmtFlags.sharpV
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:25:2: fmt.fmtFlags.precPresent
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:27:2: fmt.fmtFlags.plus
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:29:2: fmt.fmtFlags.space
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:24:2: fmt.fmtFlags.widPresent
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:26:2: fmt.fmtFlags.minus
fmt: ~/.gvm/gos/go1.9/src/fmt/format.go:28:2: fmt.fmtFlags.sharp

go get github.com/opennota/check/cmd/varcheck

varcheck -h

Usage of varcheck:
  -e	Report exported variables and constants

varcheck image/jpeg

image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/reader.go:74:2: adobeTransformYCbCr
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/reader.go:75:2: adobeTransformYCbCrK
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:54:2: quantIndexLuminance
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:55:2: quantIndexChrominance
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:91:2: huffIndexLuminanceDC
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:92:2: huffIndexLuminanceAC
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:93:2: huffIndexChrominanceDC
image/jpeg: ~/.gvm/gos/go1.9/src/image/jpeg/writer.go:94:2: huffIndexChrominanceAC

1
2
3
4
5
6
7
8
9
10
11

package main

import (
  "fmt"
  "reflect"
)

func main() {
  tag := reflect.StructTag(`species:"gopher" color:"blue"`)
  fmt.Println(tag.Get("color"), tag.Get("species"))
}

blue gopher

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

package main

import (
  "fmt"

  "github.com/fatih/structtag"
)

func main() {
  tag := `json:"foo,omitempty,string" xml:"foo"`

  // parse the tag
  tags, err := structtag.Parse(string(tag))
  if err != nil {
    panic(err)
  }

  // iterate over all tags
  for _, t := range tags.Tags() {
    fmt.Printf("tag: %+v\n", t)
  }

  // get a single tag
  jsonTag, err := tags.Get("json")
  if err != nil {
    panic(err)
  }

  // change existing tag
  jsonTag.Name = "foo_bar"
  jsonTag.Options = nil
  tags.Set(jsonTag)

  // add new tag
  tags.Set(&structtag.Tag{
    Key:     "hcl",
    Name:    "foo",
    Options: []string{"squash"},
  })

  // print the tags
  fmt.Println(tags) // Output: json:"foo_bar" xml:"foo" hcl:"foo,squash"
}

tag: json:"foo,omitempty,string"
tag: xml:"foo"
json:"foo_bar" xml:"foo" hcl:"foo,squash"

遍历语法树，展示结构体字段列表：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

package main

import (
  "fmt"
  "go/ast"
  "go/parser"
  "go/token"
)

func main() {
  src := `package main
	type Example struct {
    Foo string` + " `json:\"foo\"` }"

  fset := token.NewFileSet()
  file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
  if err != nil {
    panic(err)
  }

  ast.Inspect(file, func(x ast.Node) bool {
    s, ok := x.(*ast.StructType)
    if !ok {
      return true
    }

    for _, field := range s.Fields.List {
      fmt.Printf("Field: %s\n", field.Names[0].Name)
      fmt.Printf("Tag:   %s\n", field.Tag.Value)
    }
    return false
  })
}

Field: Foo
Tag:   `json:"foo"`

** 变量设置在Header中
   :PROPERTIES:
   :ID:       d7b197d7-f925-47b7-bcc0-37e182761619
   :header-args:shell: :var host="google.com" :results pp :session brantou :exports both
   :END:
   #+BEGIN_SRC shell
     ping $host -c 2
   #+END_SRC

   #+RESULTS:
   :
   : $ PING google.com (192.168.83.230) 56(84) bytes of data.
   : 64 bytes from 192.168.83.230: icmp_seq=1 ttl=128 time=9.60 ms
   : 64 bytes from 192.168.83.230: icmp_seq=2 ttl=128 time=25.4 ms
   :
   : --- google.com ping statistics ---
   : packet loss, time 1001ms
   : rtt min/avg/max/mdev = 9.609/17.508/25.408/7.900 ms

   覆盖变量 *host* 和 *session* 置为空！
   #+BEGIN_SRC shell :var host="baidu.com" :session none
     ping $host -c 2
   #+END_SRC

   #+RESULTS:
   : PING baidu.com (111.13.101.208) 56(84) bytes of data.
   : 64 bytes from 111.13.101.208: icmp_seq=1 ttl=128 time=35.5 ms
   : 64 bytes from 111.13.101.208: icmp_seq=2 ttl=128 time=49.2 ms
   :
   : --- baidu.com ping statistics ---
   : 2 packets transmitted, 2 received, 0% packet loss, time 1002ms
   : rtt min/avg/max/mdev = 35.517/42.361/49.206/6.847 ms

#+NAME: img_wrap
#+BEGIN_SRC python :var img_path="" :results output
  img_path=img_path.replace('[[file:..', '').replace(']]', '')
  print('<img src="'+img_path+'" />')
#+END_SRC

#+HEADER: :file ../images/devOps/just-try-post.png :exports both
#+BEGIN_SRC plantuml :mkdirp yes :post img_wrap(img_path=*this*) :wrap EXPORT html
  @startuml
  cli -> serv:  auth req
  serv --> cli: auth res
  @enduml
#+END_SRC

#+RESULTS:
#+BEGIN_EXPORT html
<img src="/images/devOps/just-try-post.png" />
#+END_EXPORT

#+HEADER: :tangle ../src/devOps/noweb_ref.sh :mkdirp yes
#+BEGIN_SRC shell :noweb yes :shebang #!/bin/sh
  <<fullest-disk>>
#+END_SRC

** the mount point of the fullest disk
   :PROPERTIES:
   :header-args:shell: :noweb-ref fullest-disk
   :END:

*** query all mounted disks
    #+BEGIN_SRC shell
        df \
    #+END_SRC
*** strip the header row
    #+BEGIN_SRC shell
        |sed '1d' \
    #+END_SRC
*** sort by the percent full
    #+BEGIN_SRC shell
        |awk '{print $5 " " $6}' | sort -n | tail -1 \
    #+END_SRC
*** extract the mount point
    #+BEGIN_SRC shell
      |awk '{print $2}'
    #+END_SRC

tangle出的代码如下：

#!/bin/sh
df \
|sed '1d' \
|awk '{print $5 " " $6}' | sort -n | tail -1 \
|awk '{print $2}'

在做相关操作时，可能需要预先初始化，同时又不想这些初始化过程出现在代码流程中，
这个时候你可使用 *:prologue* 来关联相关语句来达到预处理的效果。
#+NAME: prologue-example
#+BEGIN_SRC shell :prologue "echo yep; exit 0" :exports both
  echo "nope"
#+END_SRC

#+RESULTS: prologue-example
: yep

*:epilogue* 正好和 *:prologue* 完成的功能相反，用于后置处理，完成清理工作。
#+BEGIN_SRC shell :epilogue "exit 0; echo yep" :exports both
  echo "nope"
#+END_SRC

#+RESULTS:
: nope

可在命令行中调用代码块实现的函数，如下面所示：

#!/bin/sh
# -*- mode: shell-script -*-
#
# tangle files with org-mode
#

DIR='pwd'
FILES=""
# wrap each argument in the code required to call tangle on it
for i in $@; do
    FILES="$FILES \"$i\""
done

emacs -Q --batch \
      --eval "(progn
(require 'org)(require 'ob)(require 'ob-tangle)
(mapc (lambda (file)
       (find-file (expand-file-name file \"$DIR\"))
       (org-babel-tangle)
       (kill-buffer)) '($FILES)))" 2>&1 | grep tangled

#+BEGIN_SRC sql
  show tables;
#+END_SRC

#+RESULTS:
| Tables_in_mysql           |
|---------------------------|
| columns_priv              |
| db                        |
| engine_cost               |
| event                     |
| func                      |
| general_log               |
| gtid_executed             |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| innodb_index_stats        |
| innodb_table_stats        |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| server_cost               |
| servers                   |
| slave_master_info         |
| slave_relay_log_info      |
| slave_worker_info         |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |

#+BEGIN_SRC sql
  SELECT host, user FROM user WHERE 1;
#+END_SRC

#+RESULTS:
| host      | user             |
|-----------+------------------|
| %         | brantou          |
| localhost | debian-sys-maint |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |

这里假设 Python 的开发环境已配置完毕，若是还没配置，请自行google。

Org-mode 已内置在新的 Emacs 中，但是默认情况下， 只有 emacs-lisp 可被执行。 要启用或禁用其他语言, 可通过 Emacs 自定义交互界面来配置 org-babel-load-languages 变量, 或者将代码添加到 init 文件中, 启用python的代码如下所示:

1
2
3

(org-babel-do-load-languages
 'org-babel-load-languages
 '((python . t)))

• Hello World!

  #+begin_src python :results output
    print "Hello, world!"
  #+end_src
  
  #+RESULTS:
  : Hello, world!
  

• 参数

  #+NAME: square
  #+BEGIN_SRC python :var num=5
    def square(x):
        return x*x
  
    return square(num)
  #+END_SRC
  
  #+RESULTS: square
  : 25
  
  #+CALL: square(num=10)
  
  #+RESULTS:
  : 100
  

• 文学编程

  #+NAME: square
  #+BEGIN_SRC python
    def square(x):
        return x*x
  #+END_SRC
  
  #+NAME: calc-square
  #+BEGIN_SRC python :var num=5 :noweb strip-export :results output
    <<square>>
    print(square(num))
  #+END_SRC
  
  #+RESULTS: calc-square
  : 25
  
  #+CALL: calc-square(num=7)
  
  #+RESULTS:
  : 49
  

• 内联调用(Inline calling):

  2 加 2 等于 src_python{return(2+2)}
  

  当导出 HTML 或者 LaTeX/PDF 时，如下所示：

  2 加 2 等于 4
  

• 使用Org-mode的table作为参数

  #+tblname: data_table
  | a | 1 |
  | b | 2 |
  | c | 3 |
  #+begin_src python :var val=1 :var data=data_table
  # Return row specified by val.
  # In non-session mode, use return to return results.
  return(data[val])
  #+end_src
  
  #+RESULTS:
  | b | 2 |
  

• 绘图

  #+begin_src python :results file
    import matplotlib, numpy
    matplotlib.use('Agg')
    import matplotlib.pyplot as plt
    fig=plt.figure(figsize=(4,2))
    x=numpy.linspace(-15,15)
    plt.plot(numpy.sin(x)/x)
    fig.tight_layout()
    plt.savefig('../images/python-matplot-fig.png')
    return '../images/python-matplot-fig.png'
  #+end_src
  
  #+RESULTS:
  [[file:../images/python-matplot-fig.png]]
  

  <img src="/images/python-matplot-fig.png” />

• 词云

  #+BEGIN_SRC python :preamble "# -*- coding: utf-8 -*-" :results value file
    import jieba.analyse
    from wordcloud import WordCloud, ImageColorGenerator
    import numpy as np
    from PIL import Image
    import random
  
    font_path = '../resource/tyzkaishu.ttf'
    width = 640
    height = 480
  
    text = open('../resource/xiyouji.txt').read()
    words = jieba.analyse.extract_tags(text, topK=200, withWeight=True)
  
    word_freqs = {}
    for word in words:
        word_freqs[word[0]] = word[1]
  
    mask = np.array(Image.open('../resource/stormtrooper_mask.png'))
    wordcloud = WordCloud(
        font_path=font_path, width=width, height=height,
        mask=mask).generate_from_frequencies(word_freqs)
    wordcloud.to_file('../images/xiyouji-mask.png')
    return '../images/xiyouji-mask.png'
  #+END_SRC
  
  #+RESULTS:
  [[file:../images/xiyouji-mask.png]]
  

  <img src="/images/xiyouji-mask.png” />

当把 utf-8 的字符串传给 Python , 需要格外小心。

• Python Source Code Blocks in Org Mode
• Babel: org-mode的元编程
• 在Org-mode中执行Go代码
• Working with source code
• Babel: active code in Org-mode

在开始之前，请确保 scrpay 正确安装，手头有一款简洁而强大的浏览器， 若是你有使用 postman 那就更好了。

scrapy genspider zhihu

使用以上命令生成知乎爬虫,代码如下:

1
2
3
4
5
6
7
8
9
10
11

# -*- coding: utf-8 -*-
import scrapy


class ZhihuSpider(scrapy.Spider):
    name = 'zhihu'
    allowed_domains = ['www.zhihu.com']
    start_urls = ['http://www.zhihu.com/']

    def parse(self, response):
	pass

有一点切记，不要忘了启用 Cookies, 切记切记 ：

1
2

# Disable cookies (enabled by default)
COOKIES_ENABLED = True

过程如下：

• 进入登录页，获取 Header 和 Cookie 信息， 完善的 Header 信息能尽量伪装爬虫， 有效 Cookie 信息能迷惑知乎服务端，使其认为当前登录非首次登录，若无有效 Cookie 会遭遇验证码。 在抓取数据之前，请在浏览器中登录过知乎，这样才使得 Cookie 是有效的。

  <img src="/images/zhihu-headers-cookies.jpg” />

  Header 和 Cookie 整理如下:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

  headers = { 'Host': 'www.zhihu.com', 'Connection': 'keep-alive', 'Origin': 'https://www.zhihu.com', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Accept': '*/*', 'X-Requested-With': 'XMLHttpRequest', 'DNT': 1, 'Referer': 'https://www.zhihu.com/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6', } cookies = { 'd_c0': '"AHCAtu1iqAmPTped76X1ZdN0X_qAwhjdLUU=|1458699045"', '__utma': '51854390.1407411155.1458699046.1458699046.1458699046.1', '__utmv': '51854390.000--|3=entry_date=20160322=1', '_zap': '850897bb-cba4-4d0b-8653-fd65e7578ac2', 'q_c1': 'b7918ff9a5514d2981c30050c8c732e1|1502937247000|1491446589000', 'aliyungf_tc': 'AQAAAHVgviiNyQsAOhSntJ5J/coWYtad', '_xsrf': 'b12fdca8-cb35-407a-bc4c-6b05feff37cb', 'l_cap_id': '"MDk0MzRjYjM4NjAwNDU0MzhlYWNlODQ3MGQzZWM0YWU=|1503382513|9af99534aa22d5db92c7f58b45f3f3c772675fed"', 'r_cap_id': '"M2RlNDZjN2RkNTBmNGFmNDk2ZjY4NjIzY2FmNTE4NDg=|1503382513|13370a99ee367273b71d877de17f05b2986ce0ef"', 'cap_id': '"NmZjODUxZjQ0NzgxNGEzNmJiOTJhOTlkMTVjNWIxMDQ=|1503382513|dba2e9c6af7f950547474f827ef440d7a2950163"', }

• 在浏览器中，模拟登陆，抓取登陆请求信息。

  <img src="/images/zhihu-login-args.jpg” />

  从图中可以看到 _xsrf 参数, 这个参数与登陆验证信息无关，但很明显是由登陆页面携带的信息。 Google 了下 xsrf 的含义， 是用于防范 跨站请求伪造 。

  <img src="/images/zhihu-xsrf.jpg” />

• 整理以上，代码如下：

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

  loginUrl = 'https://www.zhihu.com/#signin' siginUrl = 'https://www.zhihu.com/login/email' def start_requests(self): return [ scrapy.http.FormRequest( self.loginUrl, headers=self.headers, cookies=self.cookies, meta={'cookiejar': 1}, callback=self.post_login) ] def post_login(self, response): xsrf = response.css( 'div.view-signin > form > input[name=_xsrf]::attr(value)' ).extract_first() self.headers['X-Xsrftoken'] = xsrf return [ scrapy.http.FormRequest( self.siginUrl, method='POST', headers=self.headers, meta={'cookiejar': response.meta['cookiejar']}, formdata={ '_xsrf': xsrf, 'captcha_type': 'cn', 'email': 'xxxxxx@163.com', 'password': 'xxxxxx', }, callback=self.after_login) ]

经过上述步骤登陆成功了，有点小激动，有没有！ 但苦难到此还远没有结束，这个时候尝试抓取最近热门话题，直接返回 code:401 ,未授权的访问。 授权信息未设置，导致了此类错误，莫非遗漏了什么，看来只能在浏览器中追踪请求参数来侦测问题。 在浏览器的请求中，包含了Bearer Token, 而我在scrapy中模拟的请求中未包含此信息，所以我被服务器认定为未授权的。 通过观察发现 Bearer Token 的关键部分，就是 Cookies 中的 z_c0 对应的信息。

z_c0 包含的信息，是在登陆完成时种下的，所以从登陆完成返回的登陆信息里，获取要设置的 Cookies 信息， 然后拼接出 Bearer Token,最后设置到 Header 中。

代码整理如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

def after_login(self, response):
    jdict = json.loads(response.body)
    print('after_login', jdict)
    if jdict['r'] == 0:
	z_c0 = response.headers.getlist('Set-Cookie')[2].split(';')[0].split(
	    '=')[1]
	self.headers['authorization'] = 'Bearer ' + z_c0
	return scrapy.http.FormRequest(
	    url=self.feedUrl,
	    method='GET',
	    meta={'cookiejar': response.meta['cookiejar']},
	    headers=self.headers,
	    formdata={
		'action_feed': 'True',
		'limit': '10',
		'action': 'down',
		'after_id': str(self.curFeedId),
		'desktop': 'true'
	    },
	    callback=self.parse)
    else:
	print(jdict['error'])

上述步骤后，数据获取就水到渠成了，为了检测成功与否， 把返回信息写到文件中,而且只获取前五十个,代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

feedUrl = 'https://www.zhihu.com/api/v3/feed/topstory'
nextFeedUrl = ''
curFeedId = 0


def parse(self, response):
    with open('zhihu.json', 'a') as fd:
	fd.write(response.body)
    jdict = json.loads(response.body)
    jdatas = jdict['data']
    for entry in jdatas:
	entry['pid'] = entry['id']
	yield entry

    jpaging = jdict['paging']
    self.curFeedId += len(jdatas)
    if jpaging['is_end'] == False and self.curFeedId < 50:
	self.nextFeedUrl = jpaging['next']
	yield self.next_request(response)


def next_request(self, response):
    return scrapy.http.FormRequest(
	url=self.nextFeedUrl,
	method='GET',
	meta={'cookiejar': response.meta['cookiejar']},
	headers=self.headers,
	callback=self.parse)

获取的数据，采用json格式， 如下所示:

知乎的数据，只有登录完成之后，才可有效的获取，所以模拟登陆是无法忽略不管的。 所谓的模拟登陆，只是在scrapy中尽量的模拟在浏览器中的交互过程，使服务端无感抓包过程。 请求中附加有效的 Cookies 和 Headers 头信息，可有效的迷惑服务端， 同时在交互的过程中，获取后续请求必要信息和认证信息，使得整个流程能不断先前。

若是你遇到什么问题，尽量提出来，欢迎一起来讨论解决。 源码放在了github上， 欢迎前往查看。 若是帮你解决了问题，或者给了你启发，不要吝啬给我加一星。