Skip to content

Transaction timeout on download #71

Description

@lwsanty

summary one of repositories downloads fails with timeout error

environment

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
GOMAXPROCS is 8
8 CPU 16G RAM

steps to reproduce
just run this command

./gitcollector download --library=/home/lwsanty/gitcollector/ --orgs=src-d,bblfsh

expected all corresponding repositories downloaded
observed one of downloads fails with timeout

[2019-09-10T16:54:41.685787348+03:00] ERROR failed error=timeout exceeded: unable to retrieve repository from location cc761bed408a1d6a20457d48d8bf807d3531db4f in transactional mode.

github.com/src-d/go-borges/siva.(*transactioner).Start
        /home/lwsanty/goproj/gopath/pkg/mod/github.com/src-d/go-borges@v0.0.0-20190704083038-44867e8f2a2a/siva/transactioner.go:45
github.com/src-d/go-borges/siva.(*Location).repository
        /home/lwsanty/goproj/gopath/pkg/mod/github.com/src-d/go-borges@v0.0.0-20190704083038-44867e8f2a2a/siva/location.go:528
github.com/src-d/go-borges/siva.(*Location).Init
        /home/lwsanty/goproj/gopath/pkg/mod/github.com/src-d/go-borges@v0.0.0-20190704083038-44867e8f2a2a/siva/location.go:242
github.com/src-d/gitcollector/downloader.PrepareRepository
        /home/lwsanty/goproj/lwsanty/gitcollector/downloader/git.go:199
github.com/src-d/gitcollector/downloader.downloadRepository
        /home/lwsanty/goproj/lwsanty/gitcollector/downloader/download.go:171
github.com/src-d/gitcollector/downloader.Download
        /home/lwsanty/goproj/lwsanty/gitcollector/downloader/download.go:78
github.com/src-d/gitcollector/library.(*Job).Process
        /home/lwsanty/goproj/lwsanty/gitcollector/library/job.go:57
github.com/src-d/gitcollector.(*worker).consumeJob.func1
        /home/lwsanty/goproj/lwsanty/gitcollector/worker.go:61
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337 id=72f28f0b-50f1-40e1-8457-e27d138525e4 job=download url=https://github.com/src-d/go-mysql-server

note issue was reproduced only once and cannot be reproduced after 3 attempts

question it looks like the problem is in transactions timeout of borges, we could avoid this problem if we propagate transaction timeout from gitcollector to borges by adding new flag to gitcollector

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions