O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Golang Project Layout and Practice

15.726 visualizações

Publicada em

This talk I will show you the golang project layout and some best practice like as the following:
1. RESTful api and GraphQL
2. Model testing (Postgres, SQLite, MySQL)
3. Software Quality
4. Data Metrics
5. Go Testing

Publicada em: Tecnologia
  • You'll notice the difference immediately when you make the switch from working with amateurs to working with professionals. I've been betting with these guys for more than three years and in that time I've made nearly £500,000! That's a life changing amount of money. If I can give you one piece of advice it's this – sign up and sign up NOW! Last time I was one of the last guys to grab a spot before Patrick closed the doors. If I hadn't gotten lucky that day I'd be half a million pounds poorer now and my life would be a hell of a lot different. 》》》 https://bit.ly/391WONw
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Don't forget another good way of simplifying your writing is using external resources (such as ⇒ www.HelpWriting.net ⇐ ). This will definitely make your life more easier
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Did you try ⇒ www.HelpWriting.net ⇐?. They know how to do an amazing essay, research papers or dissertations.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... www.EssayWrite.best - Write an essay .................................................................................................................................... www.LitReview.xyz - Summary of books .................................................................................................................................... www.Coursework.best - Online coursework .................................................................................................................................... www.Dissertations.me - proquest dissertations .................................................................................................................................... www.ReMovie.club - Movies reviews .................................................................................................................................... www.WebSlides.vip - Best powerpoint presentations .................................................................................................................................... www.WritePaper.info - Write a research paper .................................................................................................................................... www.EddyHelp.com - Homework help online .................................................................................................................................... www.MyResumeHelp.net - Professional resume writing service .................................................................................................................................. www.HelpWriting.net - Help with writing any papers ......................................................................................................................................... Save so as not to lose
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Golang Project Layout and Practice

  1. 1. Go Project Layout and Practice Bo-Yi Wu 2019.08.29 ModernWeb
  2. 2. About me • Software Engineer in Mediatek • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Teacher of Udemy Platform: Golang + Drone https://blog.wu-boy.com
  3. 3. Agenda • Go in Mediatek • Go Project Layout • Go Practices • RESTful api and GraphQL • Model testing (Postgres, SQLite, MySQL) • Software Quality • Data Metrics • Go Testing
  4. 4. Tech Stack • Initial Project using Go in 2018/01 • Golang • Easy to Learn • Performance • Deployment
  5. 5. Repository folder • api • assets • cmd • configs • docker • pkg ├── api ├── assets │   └── dist ├── cmd │   └── ggz ├── configs ├── docker │   ├── server └── pkg ├── config ├── errors ├── fixtures ├── helper ├── middleware │   ├── auth │   └── header ├── model ├── module │   ├── mailer │   ├── metrics │   └── storage ├── router │   └── routes ├── schema └── version
  6. 6. Root folder • .drone.yml (deploy config) • .revive.toml (golint config) • docker-compose.yml (DB, Redis and UI) • Makefile • go module config (go.mod and go.sum) • .env.example
  7. 7. Go Module https://blog.golang.org/using-go-modules
  8. 8. Improve Deployment Using Go Module Proxy https://github.com/gomods/athens
  9. 9. save time with proxy 97s -> 6s
  10. 10. Makefile Build, Testing, Deploy
  11. 11. GOFMT ?= gofmt "-s" GO ?= go TARGETS ?= linux darwin windows ARCHS ?= amd64 386 BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") GOFILES := $(shell find . -name "*.go" -type f) TAGS ?= sqlite sqlite_unlock_notify ifneq ($(shell uname), Darwin)   EXTLDFLAGS = -extldflags "-static" $(null) else   EXTLDFLAGS = endif ifneq ($(DRONE_TAG),)   VERSION ?= $(subst v,,$(DRONE_TAG)) else   VERSION ?= $(shell git describe --tags --always') endif
  12. 12. .env
  13. 13. GGZ_DB_DRIVER=mysql GGZ_DB_USERNAME=root GGZ_DB_PASSWORD=123456 GGZ_DB_NAME=ggz GGZ_DB_HOST=127.0.0.1:3307 GGZ_SERVER_ADDR=:8080 GGZ_DEBUG=true GGZ_SERVER_HOST=http://localhost:8080 GGZ_STORAGE_DRIVER=disk GGZ_MINIO_ACCESS_ID=xxxxxxxx GGZ_MINIO_SECRET_KEY=xxxxxxxx GGZ_MINIO_ENDPOINT=s3.example.com GGZ_MINIO_BUCKET=example GGZ_MINIO_SSL=true GGZ_AUTH0_DEBUG=true
  14. 14. docker-compose.yml
  15. 15. db: image: mysql restart: always volumes: - mysql-data:/var/lib/mysql environment: MYSQL_USER: example MYSQL_PASSWORD: example MYSQL_DATABASE: example MYSQL_ROOT_PASSWORD: example minio: image: minio/minio restart: always ports: volumes: - minio-data:/data environment: MINIO_ACCESS_KEY: minio123456 MINIO_SECRET_KEY: minio123456 command: server /data Development
  16. 16. Productionapi: image: foo/bar restart: always ports: - 8080:8080 environment: - GGZ_METRICS_TOKEN=test-prometheus-token - GGZ_METRICS_ENABLED=true labels: - "traefik.enable=true" - "traefik.basic.frontend.rule=Host:${WEB_HOST}" - "traefik.basic.protocol=http"
  17. 17. VersionCompile version info into Go binary
  18. 18. Version • -X github.com/go-ggz/ggz/pkg/ version.Version=$(VERSION) • -X github.com/go-ggz/ggz/pkg/ version.BuildDate=$(BUILD_DATE) go build -o bin/api -ldflags
  19. 19. var (   // Version number for git tag.   Version string   // BuildDate is the ISO 8601 day drone was built.   BuildDate string ) // PrintCLIVersion print server info func PrintCLIVersion() string {   return fmt.Sprintf(     "version %s, built on %s, %s",     Version,     BuildDate,     runtime.Version(),   ) }
  20. 20. BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") ifneq ($(DRONE_TAG),)   VERSION ?= $(subst v,,$(DRONE_TAG)) else   VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed 's/^v//') endif
  21. 21. AssetsEmbed files in Go https://github.com/UnnoTed/fileb0x
  22. 22. func ReadSource(origPath string) (content []byte, err error) {   content, err = ReadFile(origPath)   if err != nil {     log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath)   }   if config.Server.Assets != "" && file.IsDir(config.Server.Assets) {     origPath = path.Join(config.Server.Assets, origPath)     if file.IsFile(origPath) {       content, err = ioutil.ReadFile(origPath)       if err != nil {         log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath)       }     }   }   return content, err } Debug Setting
  23. 23. // ViewHandler support dist handler from UI func ViewHandler() gin.HandlerFunc {   fileServer := http.FileServer(dist.HTTP)   data := []byte(time.Now().String())   etag := fmt.Sprintf("%x", md5.Sum(data))   return func(c *gin.Context) {     c.Header("Cache-Control", "public, max-age=31536000")     c.Header("ETag", etag)     if match := c.GetHeader("If-None-Match"); match != "" {       if strings.Contains(match, etag) {         c.Status(http.StatusNotModified)         return       }     }     fileServer.ServeHTTP(c.Writer, c.Request)   } } File Server Handler
  24. 24. 圖片來來源:https://developers.google.com/web/fundamentals/performance/optimizing- content-efficiency/http-caching?hl=zh-tw
  25. 25. // Favicon represents the favicon. func Favicon(c *gin.Context) {   file, _ := dist.ReadFile("favicon.ico")   etag := fmt.Sprintf("%x", md5.Sum(file))   c.Header("ETag", etag)   c.Header("Cache-Control", "max-age=0")   if match := c.GetHeader("If-None-Match"); match != "" {     if strings.Contains(match, etag) {       c.Status(http.StatusNotModified)       return     }   }   c.Data(     http.StatusOK,     "image/x-icon",     file,   ) } NO Cache
  26. 26. API
  27. 27. /healthz • health check for load balancer func Heartbeat(c *gin.Context) {   c.AbortWithStatus(http.StatusOK)   c.String(http.StatusOK, "ok") }
  28. 28. CMDCommand line
  29. 29. Command line package • Golang package: flag • urfave/cli • spf13/cobra
  30. 30. ├── agent │   ├── config │   │   └── config.go │   └── main.go ├── notify │   └── main.go └── tcp-server ├── config │   └── config.go └── main.go
  31. 31. Config Management github.com/spf13/viper
  32. 32. Config management • Load config from File • .json • .ini • Load config from Environment Variables • .env
  33. 33.   var envfile string flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment variables")   flag.Parse()   godotenv.Load(envfile) _ "github.com/joho/godotenv/autoload"
  34. 34.   Logging struct {     Debug bool `envconfig:"GGZ_LOGS_DEBUG"`     Level string `envconfig:"GGZ_LOGS_LEVEL" default:"info"`     Color bool `envconfig:"GGZ_LOGS_COLOR"`     Pretty bool `envconfig:"GGZ_LOGS_PRETTY"`     Text bool `envconfig:"GGZ_LOGS_TEXT"`   }   // Server provides the server configuration.   Server struct {     Addr string `envconfig:"GGZ_SERVER_ADDR"`     Port string `envconfig:"GGZ_SERVER_PORT" default:"12000"`     Path string `envconfig:”GGZ_SERVER_PATH" default:"data"`   } github.com/kelseyhightower/envconfig
  35. 35.   config, err := config.Environ()   if err != nil {     log.Fatal().       Err(err).       Msg("invalid configuration")   }   initLogging(config)   // check folder exist   if !file.IsDir(config.Server.Path) {     log.Fatal().       Str("path", config.Server.Path).       Msg("log folder not found")   } Load env from structure
  36. 36. /configs Configuration file templates or default config
  37. 37. global: scrape_interval: 5s external_labels: monitor: 'my-monitor' scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'ggz-server' static_configs: - targets: ['ggz-server:8080'] bearer_token: 'test-prometheus-token'
  38. 38. /docker Docker file template
  39. 39. ├── ggz-redirect │ ├── Dockerfile.linux.amd64 │ ├── Dockerfile.linux.arm │ ├── Dockerfile.linux.arm64 │ ├── Dockerfile.windows.amd64 │ └── manifest.tmpl └── ggz-server ├── Dockerfile.linux.amd64 ├── Dockerfile.linux.arm ├── Dockerfile.linux.arm64 ├── Dockerfile.windows.amd64 └── manifest.tmpl
  40. 40. /integrations
  41. 41.   ctx := context.Background()   req := testcontainers.ContainerRequest{     Image: "goggz/ggz-server",     ExposedPorts: []string{"8080/tcp"},     WaitingFor: wait.ForLog("Starting shorten server on :8080")   }   ggzServer, err := testcontainers.GenericContainer(     ctx,     testcontainers.GenericContainerRequest{       ContainerRequest: req,       Started: true,     })   if err != nil {     t.Fatal(err)   } github.com/testcontainers/testcontainers-go
  42. 42. /pkg
  43. 43. ├── config ├── errors ├── fixtures ├── helper ├── middleware │ ├── auth │ └── header ├── model ├── module │ ├── metrics │ └── storage │ ├── disk │ └── minio ├── router │ └── routes ├── schema └── version
  44. 44. /pkg/errors
  45. 45. // Type defines the type of an error type Type string const (   // Internal error   Internal Type = "internal"   // NotFound error means that a specific item does not exis   NotFound Type = "not_found"   // BadRequest error   BadRequest Type = "bad_request"   // Validation error   Validation Type = "validation"   // AlreadyExists error   AlreadyExists Type = "already_exists"   // Unauthorized error   Unauthorized Type = "unauthorized" )
  46. 46. // ENotExists creates an error of type NotExist func ENotExists(msg string, err error, arg ...interface{}) error {   return New(NotFound, fmt.Sprintf(msg, arg...), err) } // EBadRequest creates an error of type BadRequest func EBadRequest(msg string, err error, arg ...interface{}) error {   return New(BadRequest, fmt.Sprintf(msg, arg...), err) } // EAlreadyExists creates an error of type AlreadyExists func EAlreadyExists(msg string, err error, arg ...interface{}) error {   return New(AlreadyExists, fmt.Sprintf(msg, arg...), err) }
  47. 47. /pkg/fixtures
  48. 48. Rails-like test fixtures Write tests against a real database github.com/go-testfixtures/testfixtures
  49. 49. fixtures/ posts.yml comments.yml tags.yml posts_tags.yml users.yml
  50. 50. - id: 1 email: test@gmail.com full_name: test avatar: http://example.com avatar_email: test@gmail.com - id: 2 email: test1234@gmail.com full_name: test1234 avatar: http://example.com avatar_email: test1234@gmail.com
  51. 51. Unit Testing with Database
  52. 52. func TestMain(m *testing.M) { // test program to do extra setup or teardown before or after testing. os.Exit(m.Run()) } https://golang.org/pkg/testing/#hdr-Main
  53. 53. func MainTest(m *testing.M, pathToRoot string) {   var err error   fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures")   if err = createTestEngine(fixturesDir); err != nil {     fatalTestError("Error creating test engine: %vn", err)   }   os.Exit(m.Run()) } func createTestEngine(fixturesDir string) error {   var err error   x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared")   if err != nil {     return err   }   x.ShowSQL(config.Server.Debug)   return InitFixtures(&testfixtures.SQLite{}, fixturesDir) } Testing with SQLite
  54. 54. func TestIsUserExist(t *testing.T) {   assert.NoError(t, PrepareTestDatabase())   exists, err := IsUserExist(0, "test@gmail.com")   assert.NoError(t, err)   assert.True(t, exists)   exists, err = IsUserExist(0, "test123456@gmail.com")   assert.NoError(t, err)   assert.False(t, exists)   exists, err = IsUserExist(1, "test1234@gmail.com")   assert.NoError(t, err)   assert.True(t, exists)   exists, err = IsUserExist(1, "test123456@gmail.com")   assert.NoError(t, err)   assert.False(t, exists) } go test -v -run=TestIsUserExist ./pkg/models/
  55. 55. /pkg/helper
  56. 56. Helper func • Encrypt and Decrypt • Regexp func • IsEmail, IsUsername • Zipfile
  57. 57. /pkg/middleware
  58. 58. func Secure(c *gin.Context) {   c.Header("Access-Control-Allow-Origin", "*")   c.Header("X-Frame-Options", "DENY")   c.Header("X-Content-Type-Options", "nosniff")   c.Header("X-XSS-Protection", "1; mode=block")   if c.Request.TLS != nil {     c.Header("Strict-Transport-Security", "max-age=31536000")   } }
  59. 59. /pkg/model
  60. 60. Use gorm or xorm
  61. 61. Build in SQLite3
  62. 62. // +build sqlite package model import (   _ "github.com/mattn/go-sqlite3" ) func init() {   EnableSQLite3 = true } go build -v -tags 'sqlite sqlite_unlock_notify'
  63. 63. /pkg/module
  64. 64. ├── cron ├── download ├── jwt ├── ldap ├── mailer │ ├── ses │ └── smtp ├── queue ├── metrics ├── redis └── storage ├── disk └── minio
  65. 65. Integration with Prometheus + Grafana
  66. 66. func NewCollector() Collector {   return Collector{     Users: prometheus.NewDesc(       namespace+"users",       "Number of Users",       nil, nil,     ), } // Collect returns the metrics with values func (c Collector) Collect(ch chan<- prometheus.Metric) {   stats := model.GetStatistic()   ch <- prometheus.MustNewConstMetric(     c.Users,     prometheus.GaugeValue,     float64(stats.Counter.User),   ) }
  67. 67. Prometheus Handler
  68. 68. func Metrics(token string) gin.HandlerFunc {   h := promhttp.Handler()   return func(c *gin.Context) {     if token == "" {       h.ServeHTTP(c.Writer, c.Request)       return     }     header := c.Request.Header.Get("Authorization")     if header == "" {       c.String(http.StatusUnauthorized, errInvalidToken.Error())       return     }     bearer := fmt.Sprintf("Bearer %s", token)     if header != bearer {       c.String(http.StatusUnauthorized, errInvalidToken.Error())       return     }     h.ServeHTTP(c.Writer, c.Request)   } }
  69. 69.     c := metrics.NewCollector()     prometheus.MustRegister(c)     if config.Metrics.Enabled {       root.GET("/metrics", router.Metrics(config.Metrics.Token))     } Your prometheus token
  70. 70. /pkg/schema
  71. 71. RESTful vs GraphQL See the Slide: GraphQL in Go
  72. 72. var rootQuery = graphql.NewObject(   graphql.ObjectConfig{     Name: "RootQuery",     Description: "Root Query",     Fields: graphql.Fields{       "queryShortenURL": &queryShortenURL,       "queryMe": &queryMe,     },   }) var rootMutation = graphql.NewObject(   graphql.ObjectConfig{     Name: "RootMutation",     Description: "Root Mutation",     Fields: graphql.Fields{       "createUser": &createUser,     },   }) // Schema is the GraphQL schema served by the server. var Schema, _ = graphql.NewSchema(   graphql.SchemaConfig{     Query: rootQuery,     Mutation: rootMutation,   })
  73. 73. Write the GraphQL Testing
  74. 74.   assert.NoError(t, model.PrepareTestDatabase())   t.Run("user not login", func(t *testing.T) {     test := T{       Query: `{ queryMe { email } }`,       Schema: Schema,       Expected: &graphql.Result{         Data: map[string]interface{}{           "queryMe": nil,         },         Errors: []gqlerrors.FormattedError{           {             Message: errorYouAreNotLogin,           },         },       },     }   }) }
  75. 75. Best Practice Testing your Go code
  76. 76. Testable Code • Code Quality • Readability • Maintainability • Testability
  77. 77. #1. Testing in Go func TestFooBar(t *testing.T) {} func ExampleFooBar(t *testing.T) {} func BenchmarkFooBar(b *testing.B) {} go test package_name
  78. 78. #2. Benchmark Testing Profiling: CPU, Memory, Goroutine Block
  79. 79. func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) {   for i := 0; i < b.N; i++ {     context := map[string]interface{}{}     variables := map[string]interface{}{}     playlyfeExecutor.Execute(context, "{hello}", variables, "")   } } func BenchmarkGophersGraphQLMaster(b *testing.B) {   for i := 0; i < b.N; i++ {     ctx := context.Background()     variables := map[string]interface{}{}     gopherSchema.Exec(ctx, "{hello}", "", variables)   } } http://bit.ly/2L0CG3Q
  80. 80. #3. Example Testing Examples on how to use your code
  81. 81. func ExampleFooBar() {   fmt.Println(strings.Compare("a", "b"))   fmt.Println(strings.Compare("a", "a"))   fmt.Println(strings.Compare("b", "a"))   // Output:   // -1   // 0   // 1 }
  82. 82. $ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/... === RUN ExampleFooBar --- PASS: ExampleFooBar (0.00s) PASS ok github.com/go-ggz/ggz/pkg/model 0.022s
  83. 83. #4. Subtests in Testing Package func (t *T) Run(name string, f func(t *T)) bool {} func (b *B) Run(name string, f func(b *B)) bool {}
  84. 84.   tests := []struct {     name string     fields fields     args args   }{}   for _, tt := range tests {     t.Run(tt.name, func(t *testing.T) {       c := Collector{         Shortens: tt.fields.Shortens,         Users: tt.fields.Users,       }       c.Describe(tt.args.ch)     })   } }
  85. 85. #5. Skipping Testing t.Skip()
  86. 86. package metrics import (   "os"   "testing" ) func TestSkip(t *testing.T) {   if os.Getenv("DEBUG_MODE") == "true" {     t.Skipf("test skipped")   } }
  87. 87. #6. Running Tests in Parallel Speedup your CI/CD Flow t.Parallel()
  88. 88. func TestFooBar01(t *testing.T) {   t.Parallel()   time.Sleep(time.Second) } func TestFooBar02(t *testing.T) {   t.Parallel()   time.Sleep(time.Second * 2) } func TestFooBar03(t *testing.T) {   t.Parallel()   time.Sleep(time.Second * 3) }
  89. 89. Just only use one package github.com/stretchr/testify
  90. 90. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG2019
  91. 91. https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019
  92. 92. END

×