Infinite loop in RefreshTokenManager

[MasseGuillaume]

I'm using akka-http-session 0.4.0 in scastie https://github.com/scalacenter/scastie/blob/master/server/src/main/scala/com.olegych.scastie.web/oauth2/GithubUserSession.scala

I have two threads at 100% cpu usage. This the result of jstack:

"Web-akka.actor.default-dispatcher-214" #273 prio=5 os_prio=0 tid=0x00007f69fc024000 nid=0x33ba runnable [0x00007f69bfbf8000]
   java.lang.Thread.State: RUNNABLE
        at scala.collection.mutable.HashTable.removeEntry(HashTable.scala:193)
        at scala.collection.mutable.HashTable.removeEntry$(HashTable.scala:181)
        at scala.collection.mutable.HashMap.removeEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.remove(HashMap.scala:123)
        at com.softwaremill.session.InMemoryRefreshTokenStorage.remove(RefreshTokenStorage.scala:59)
        at com.softwaremill.session.InMemoryRefreshTokenStorage.remove$(RefreshTokenStorage.scala:57)
        at com.olegych.scastie.web.oauth2.GithubUserSession$$anon$1.remove(GithubUserSession.scala:52)
        at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$4(SessionManager.scala:130)
        at com.softwaremill.session.RefreshTokenManager$$Lambda$1909/515478046.apply(Unknown Source)
        at com.softwaremill.session.InMemoryRefreshTokenStorage.schedule(RefreshTokenStorage.scala:64)
        at com.softwaremill.session.InMemoryRefreshTokenStorage.schedule$(RefreshTokenStorage.scala:62)
        at com.olegych.scastie.web.oauth2.GithubUserSession$$anon$1.schedule(GithubUserSession.scala:52)
        at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$3(SessionManager.scala:130)
        at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$3$adapted(SessionManager.scala:127)
        at com.softwaremill.session.RefreshTokenManager$$Lambda$1736/616177409.apply(Unknown Source)
        at scala.Option.foreach(Option.scala:257)
 at com.softwaremill.session.RefreshTokenManager.rotateToken(SessionManager.scala:127)
        at com.softwaremill.session.RefreshTokenManager.rotateToken$(SessionManager.scala:115)
        at com.softwaremill.session.SessionManager$$anon$3.rotateToken(SessionManager.scala:24)
        at com.softwaremill.session.RefreshableSessionDirectives.$anonfun$setRefreshToken$1(SessionDirectives.scala:202)
        at com.softwaremill.session.RefreshableSessionDirectives$$Lambda$1731/488540775.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive$SingleValueModifiers.$anonfun$flatMap$1(Directive.scala:141)
        at akka.http.scaladsl.server.Directive$SingleValueModifiers$$Lambda$616/1043639521.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$tflatMap$2(Directive.scala:69)
        at akka.http.scaladsl.server.Directive$$Lambda$625/913242382.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$tmap$2(Directive.scala:63)
        at akka.http.scaladsl.server.Directive$$Lambda$624/2117642238.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:61)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1028/41947542.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1.$anonfun$directive$3(FutureDirectives.scala:97)
        at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1$$Lambda$1583/151472226.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:55)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1.$anonfun$directive$2(FutureDirectives.scala:97)
        at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1$$Lambda$1580/1652741769.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
 at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$5(Directive.scala:94)
        at akka.http.scaladsl.server.Directive$$Lambda$1135/82863731.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
        at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$1016/771814596.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
 at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1070/2000999693.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$1016/771814596.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:67)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1015/218875704.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$878/485475507.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:79)
        at akka.http.scaladsl.server.Route$$$Lambda$879/1869247587.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsync$$anon$23.onPush(Ops.scala:1172)
 at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
        at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:728)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:743)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[adamw]

Is this on production? I see that you are using InMemoryRefreshTokenStorage, which is rather meant for tests, especially that it can cause race conditions with multiple concurrent HTTP requests. Plus after server restart all tokens are lost :). So my first guess would be to try using a database-backed refresh token storage (or remove refresh tokens altogether, if you don't need them).

And possibly better documenting InMemoryRefreshTokenStorage :)

[MasseGuillaume]

@adamw yep in production!

I back them up in a file for persistence.

[adamw]

Ah :) Would be good to know if the map maybe grows too big, causing pressure on the GC? Maybe you can see how much memory the app has and how much is free?

If you are using InMemoryRefreshTokenStorage, you should also properly implement schedule to avoid the race conditions :)