v3.0.0 release

Author: asmyasnikov

.github/workflows/changelog.yml

@@ -0,0 +1,19 @@
+on:
+  pull_request_target:
+
+name: changelog
+
+jobs:
+  changelog:
+    name: changelog
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v2
+
+      - name: Changelog updated
+        uses: Zomzog/[email protected]
+        with:
+          fileName: CHANGELOG.md
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/version.yml

@@ -0,0 +1,19 @@
+on:
+  pull_request_target:
+
+name: version
+
+jobs:
+  changelog:
+    name: version
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v2
+
+      - name: Version updated
+        uses: Zomzog/[email protected]
+        with:
+          fileName: internal/meta/version.go
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

AUTHORS

@@ -1,4 +1,4 @@
 The following authors have created the source code of "Yandex Database GO SDK"
 published and distributed by YANDEX LLC as the owner:
 
-Sergey Kamardin <kamardin@yandex-team.ru>
+Aleksey Myasnikov <asmyasnikov@yandex-team.ru>

CHANGELOG.md

@@ -1,3 +1,16 @@
+## 3.0.0
+* Refactored sources for splitting public interfaces and internal
+  implementation for core changes in the future without change major version
+* Refactored of transport level of driver - now we use grpc code generation by stock `protoc-gen-go` instead internal protoc codegen. New API provide operate from codegen grpc-clients with driver as a single grpc client connection. But driver hide inside self a pool of grpc connections to different cluster endpoints YDB. All communications with YDB (base services includes to driver: table, discovery, coordiantion and ratelimiter) provides stock codegen grpc-clients now.
+* Much changed API of driver for easy usage.
+* Dropped package `ydbsql` (moved to external project)
+* Extracted yandex-cloud authentication to external project
+* Extracted examples to external project
+* Changed of traces API for next usage in jaeger и prometheus
+* Dropped old APIs marked as `deprecated`
+* Added integration tests with docker ydb container
+* Changed table session and endpoint link type from string address to integer NodeID
+
 ## 2.10.1
 * Fixed race on ydbsql concurrent connect. This hotfix only for v2 version
 

NEXT_MAJOR_RELEASE.md

@@ -1,30 +1,2 @@
 # Breaking changes for the next major release
-- [x] Delete deprecated api package
-- [x] Delete deprecated internalapi package
-- [x] Delete deprecated parameter KeepAliveBatchSize from session pool
-- [x] Delete deprecated ConnUsePolicy, EndpointInfo, WithEndpointInfo, WithEndpointInfoAndPolicy, ContextConn
-- [x] Delete deprecated client option DefaultMaxQueryCacheSize, MaxQueryCacheSize and client query cache
-- [x] Change `proto` codegen code in `api` from `internal/cmd/protoc-gen` tool to standard `protoc-gen-go` tool. 
-  This need for change imports to standard. Current imports are deprecated and linters alarms
-- [x] Replace grpc and protobuf libraries to actual
-- [x] Replace all internal usages of `driver.Call()` and `driver.StreamRead()` to code-generated grpc-clients,
-      which will be use driver as `grpc.ClientConnInterface`  provider.
-- [x] Delete deprecated Driver interface
-- [x] Remove or hide (do private) deprecated API for new `scanner`.
-- [x] Delete deprecated ready statistics from session pool
-- [x] Hide (do private) entity `table.Client` or `table.SessionPool` because it most difficultly for SDK users
-- [x] Drop `table.SessionProvider.PutBusy()` interface func
-- [x] Drop `Retry.MustCheckSession()` func
-- [x] Drop `RetryCheckSession` constant
-- [x] Drop `table.SessionPoolStats.BusyCheck` counter
-- [x] Drop `ydbsql.WithSessionPoolBusyCheckInterval()`
-- [x] Drop `connect.WithSessionPoolBusyCheckInterval()`
-- [x] Drop marked as deprecated some retry constants
-- [x] Extract auth package to neighbour project(-s) for isolation ydb-go-sdk from unnecessary dependencies
-- [x] Extract coordination package to neighbour project as plugin over ydb-go-sdk
-- [x] Extract ratelimiter package to neighbour project as plugin over ydb-go-sdk
-- [x] Extract experimental package to neighbour project as plugin over ydb-go-sdk
-- [x] Refactoring Trace API: exclude duplicates of data from closure Trace functions
-- [x] Move traceutil from internal to root
-- [X] Drop ReadConnStats() public method. Use Cluster.Stats() instead
-- [x] Drop deprecated NextStreamSet (merged logic into NextSet)
+- [ ] 

README.md

@@ -59,17 +59,12 @@ The straightforward example of querying data may look similar to this:
    // Determine timeout for connect or do nothing
    ctx := context.Background()
 
-   connectParams, err := ydb.ConnectionString(os.Getenv("YDB"))
-   if err != nil {
-   _, _ = fmt.Fprintf(os.Stderr, "cannot create connect params from connection string env['YDB'] = '%s': %v\n", os.Getenv("YDB"), err)
-   os.Exit(1)
-   }
-   
    // connect package helps to connect to database, returns connection object which
    // provide necessary clients such as table.Client, scheme.Client, etc.
    db, err := ydb.New(
       ctx,
       connectParams,
+	  ydb.WithConnectionString(ydb.ConnectionString(os.Getenv("YDB_CONNECTION_STRING"))),
       ydb.WithDialTimeout(3 * time.Second),
       ydb.WithCertificatesFromFile("~/.ydb/CA.pem"),
       ydb.WithSessionPoolIdleThreshold(time.Second * 5),
@@ -80,35 +75,42 @@ The straightforward example of querying data may look similar to this:
    if err != nil {
       // handle error
    }
-   defer func() { _ = db.Close() }()
-   
-   // Create session for execute queries
-   session, err := db.Table().CreateSession(ctx)
-   if err != nil {
-       // handle error
-   }
-   defer session.Close(ctx)
+   defer func() { _ = db.Close(ctx) }()
 
    // Prepare transaction control for upcoming query execution.
    // NOTE: result of TxControl() may be reused.
    txc := table.TxControl(
-       table.BeginTx(table.WithSerializableReadWrite()),
-       table.CommitTx(),
+      table.BeginTx(table.WithSerializableReadWrite()),
+      table.CommitTx(),
    )
 
-   // Execute text query without preparation and with given "autocommit"
-   // transaction control. That is, transaction will be committed without
-   // additional calls. Notice the "_" unused variable – it stands for created
-   // transaction during execution, but as said above, transaction is committed
-   // for us and `ydb-go-sdk` do not want to do anything with it.
-   _, res, err := session.Execute(ctx, txc,
-       `--!syntax_v1
-           DECLARE $mystr AS Utf8?;
-           SELECT 42 as id, $mystr as mystr
-       `,
-       table.NewQueryParameters(
-           table.ValueParam("$mystr", types.OptionalValue(types.UTF8Value("test"))),
-       ),
+   var res *table.Result
+
+   // Do() provide the best effort for executing operation
+   // Do implements internal busy loop until one of the following conditions occurs:
+   // - deadline was cancelled or deadlined
+   // - operation returned nil as error
+   // Note that in case of prepared statements call to Prepare() must be made
+   // inside the function body.
+   err := c.Do(
+      ctx, 
+      func(ctx context.Context, s table.Session) (err error) {
+         // Execute text query without preparation and with given "autocommit"
+         // transaction control. That is, transaction will be committed without
+         // additional calls. Notice the "_" unused variable – it stands for created
+         // transaction during execution, but as said above, transaction is committed
+         // for us and `ydb-go-sdk` do not want to do anything with it.
+         _, res, err := session.Execute(ctx, txc,
+            `--!syntax_v1
+             DECLARE $mystr AS Utf8?;
+             SELECT 42 as id, $mystr as mystr
+             `,
+            table.NewQueryParameters(
+               table.ValueParam("$mystr", types.OptionalValue(types.UTF8Value("test"))),
+            ),
+         )
+         return err
+      },
    )
    if err != nil {
        return err // handle error
@@ -139,35 +141,13 @@ The straightforward example of querying data may look similar to this:
    }
 ```
 
-This example can be tested as https://github.com/ydb-platform/ydb-go-examples/tree/master/from_readme
-
 YDB sessions may become staled and appropriate error will be returned. To
-reduce boilerplate overhead for such cases `ydb-go-sdk` provides generic retry logic:
-
-```go
- var res *table.Result
- // Retry() provide the best effort fo retrying operation
- // Retry implements internal busy loop until one of the following conditions occurs:
- // - deadline was cancelled or deadlined
- // - retry operation returned nil as error
- // Note that in case of prepared statements call to Prepare() must be made
- // inside the function body.
- err := c.Retry(ctx, false,
-     func(ctx context.Context, s table.Session) (err error) {
-         res, err = s.Execute(...)
-         return
-     },
- )
-```
-
-That is, instead of manual creation of `table.Session`, we give a
-`table.Client` such responsibility. It holds instances of active sessions and
-"pings" them periodically to keep them alive.
+reduce boilerplate overhead for such cases `ydb-go-sdk` provides generic retry logic
 
 ## Credentials <a name="Credentials"></a>
 
 There are different variants to get `ydb.Credentials` object to get authorized.
-Usage examples can be found [here](https://github.com/ydb-platform/ydb-go-examples/tree/master/auth).
+Usage examples can be found [here](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth).
 
 ## Examples <a name="examples"></a>
 

go.mod

@@ -3,7 +3,15 @@ module github.com/ydb-platform/ydb-go-sdk/v3
 go 1.14
 
 require (
+	github.com/ydb-platform/ydb-api-protos v0.0.0-20210921091122-f697ac767e19 // indirect
 	github.com/ydb-platform/ydb-go-genproto v0.0.0-20210916081217-f4e55570b874
-	google.golang.org/grpc v1.37.0
+	google.golang.org/grpc v1.38.0
 	google.golang.org/protobuf v1.26.0
 )
+
+retract (
+	v3.0.0
+	v3.0.1
+	v3.0.2
+	v3.0.3
+)

go.sum

@@ -33,6 +33,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
+github.com/ydb-platform/ydb-api-protos v0.0.0-20210921091122-f697ac767e19 h1:uywQn4qvfAgy7muv7G+C8t+KbKnTkQhs82O45/9CcPg=
+github.com/ydb-platform/ydb-api-protos v0.0.0-20210921091122-f697ac767e19/go.mod h1:I0NWnA2hX8d2tKctW3AO2ne1Xc+FhvhA8PbP4iXQa5M=
 github.com/ydb-platform/ydb-go-genproto v0.0.0-20210916081217-f4e55570b874 h1:wbpYXEn87uEDkH3RvG37XW2ipEkqxpJUCGAoU1on0yE=
 github.com/ydb-platform/ydb-go-genproto v0.0.0-20210916081217-f4e55570b874/go.mod h1:cc138nptTn9eKptCQl/grxP6pBKpo/bnXDiOxuVZtps=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
@@ -70,8 +72,9 @@ google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZi
 google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
 google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
 google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
-google.golang.org/grpc v1.37.0 h1:uSZWeQJX5j11bIQ4AJoj+McDBo29cY1MCoC1wO3ts+c=
 google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
+google.golang.org/grpc v1.38.0 h1:/9BgsAsa5nWe26HqOlvlgJnqBuktYOLCgjCPqsa56W0=
+google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=

internal/driver/cluster/balancer/conn/conn.go

@@ -1,7 +1,5 @@
 package conn
 
-import "C"
-
 import (
 	"context"
 	"sync"

internal/meta/version.go

@@ -1,5 +1,5 @@
 package meta
 
 const (
-	Version = "ydb-go-sdk/2.10.1"
+	Version = "ydb-go-sdk/3.0.0"
 )