Feat Breakdowns #10613

0xpeluche · 2025-09-15T09:25:26Z

No description provided.

g1nt0ki · 2025-09-15T11:28:14Z

defi/src/adaptors/db-utils/db2.ts

    const batch: any = await Tables.DIMENSIONS_DATA.findAll({
      where: { type: adapterType, updatedat: { [Op.gte]: timestamp * 1000 } },
-      attributes: ['data', 'timestamp', 'id', 'timeS'],
+      attributes: ['data', 'timestamp', 'id', 'timeS', 'bl', 'blc'],


blc are being read, but as long as this variable is present DIM_STORE_BLC === 'false' in the code, they are not stored in the common cache file. This allows for an easy switch if one day you want to enable them, but if you prefer I can completely remove blc related logic so that it no longer appears in the code at all

g1nt0ki · 2025-09-15T11:28:31Z

defi/src/adaptors/db-utils/db2.ts

    const batch: any = await Tables.DIMENSIONS_DATA.findAll({
      where: filterCondition,
-      attributes: ['data', 'timestamp', 'id', 'timeS'],
+      attributes: ['data', 'timestamp', 'id', 'timeS', 'bl', 'blc'],


g1nt0ki · 2025-09-15T22:04:54Z

defi/src/api2/cron-task/dimensions.ts

 }

-
+const STORE_BLC = process.env.DIM_STORE_BLC === 'false'


try pulling this from the env.ts file, this way, we have a record of all the flags that we use

on second thought, can you remove all blc related code for now? also, and summary breakdown code as well, lets focus only on financial statements, so, only for fee data:

we aggregate all data of given metric by month & by quarter. (df, dr, dssr, dhr, etc), also for the labels if present by metic

add a smart logic that adds missing data where possible, like if df & dr are present, show aggregated dssr as df - dr (and other way around), same for (dhr = dr - dssr)

add breakdown labels as part of final output file

probably, best to generate parent protocol data after the child data is generated, easier to read and merge aggregated child data?

g1nt0ki · 2025-09-15T22:07:48Z

defi/src/api2/routes/index.ts

  router.get("/overview/:type", ew(getOverviewFileRoute))
  router.get("/overview/:type/:chain", ew(getOverviewFileRoute))
  router.get("/summary/:type/:name", ew(getDimensionProtocolFileRoute))
+  router.get("/summaryBreakdown/:type/:name", ew(getDimensionBreakdownProtocolFileRoute))


what I would do here is, instead of adding these under normal router, will create a new pro api router and add it there

g1nt0ki

.

g1nt0ki · 2025-09-21T09:40:11Z

defi/src/api2/cron-task/dimensions.ts

+  return filename
+}
+
+function findBuildDir(): string | null {


no, this is the wrong approach, if you want to read a generated file, you should use readRouteFile

*readRouteData

g1nt0ki · 2025-09-21T09:43:59Z

defi/src/api2/cron-task/dimensions.ts

+
+    for (const { abbr, dir } of recordTypeDirs) {
+      let files: string[] = []
+      try { files = fs.readdirSync(dir) } catch { files = [] }


no, you can get this from 'fees/df-lite' and read protocol list from there? or, get fee protocol adapter list and try to read each adapter data from there?

g1nt0ki · 2025-09-22T12:09:17Z

defi/src/api2/routes/index.ts

 }

+export function setProRoutes(router: HyperExpress.Router, _routerBasePath: string) {
+  router.get("/summaryBreakdown/:type/:name", ew(getDimensionBreakdownProtocolFileRoute))


can you change this to /v2/financial-statement/:name, for now, lets restrict to aggregated monthly/quartly of combined fee data

g1nt0ki · 2025-09-22T12:12:51Z

defi/src/api2/routes/index.ts

+  router.get("/aggregates/:type/:name", ew(getAggregatesProtocolFileRoute))
+  router.get("/aggregatesBreakdown/:type/:name", ew(getAggregatesBreakdownProtocolFileRoute))
+}
+


can we combine these two to /v2/metrics/:type/:name?breakdown=true|false
adding versioning to make it easy for us to deprecate api in the future, and breakdown as request parameter to cut down on the number of exposed routes

g1nt0ki · 2025-09-22T12:14:57Z

defi/src/api2/routes/dimensions.ts

+  const data = await readRouteData(routeFile)
+  if (!data) return errorResponse(res, errorMessage)
+
+  const response: any = { ...data }


no need to clone since we are reading from file each time?

g1nt0ki · 2025-09-22T12:15:40Z

defi/src/api2/routes/dimensions.ts

+
+  const filteredMetrics: any = {}
+  Object.keys(data.metrics || {}).forEach(key => {
+    const isBreakdownMetric = key.endsWith('bl')


hmm, need to check how the file is generated, my idea with files was, they can more or less streamed as response without any processing

g1nt0ki · 2025-09-22T12:15:54Z

defi/src/api2/routes/dimensions.ts

+
+  response.metrics = filteredMetrics
+
+  const cleanResponse = JSON.parse(JSON.stringify(response))


not needed?

g1nt0ki · 2025-09-22T12:19:50Z

defi/src/api2/routes/dimensions.ts

+export async function getDimensionBreakdownProtocolFileRoute(req: HyperExpress.Request, res: HyperExpress.Response) {
+  const protocolName = req.path_parameters.name?.toLowerCase()


can you add validation that adapterType is fee here, and since it is monthly/quarterly report, I want it to include all datatypes, this way, when the front-end shows financial report of the given protocol it fetches everything in a single call

g1nt0ki · 2025-09-22T12:21:24Z

defi/src/api2/cron-task/dimensions.ts

+  if (!src) return
+  target.metadata  = target.metadata  || {}
+
+  const KEYS = [


hmm, this exact list is used somewhere else, better to move that list to the top and this way, it is not duplicated

g1nt0ki · 2025-09-22T12:40:41Z

defi/src/api2/cron-task/dimensions.ts

+
+    Object.entries(bl).forEach(([label, v]: any) => {
+      const n = +v || 0
+      byLabel[label] = (byLabel[label] ?? 0) + n


if a value is 0 shouldnt we skip that label entry? and after this step, if the entire byLabel empty, shouldnt we skip that datapoint?

I am in two minds how to handle the parent protcol labels, it has the aggregated protocol data, did you add logic to merge bl field as well?

g1nt0ki · 2025-09-22T12:46:15Z

defi/src/api2/cron-task/dimensions.ts

      })

+      // backfill full history ONLY for protocols that newly emitted bl and have not been backfilled
+      const toBackfill = Array.from(protocolsWithBlDataInRecentUpdates)


no, it is much simpler to reset the cache once than do this, please remove this block of code

g1nt0ki · 2025-09-22T12:54:39Z

defi/src/api2/cron-task/dimensions.ts

+
+    const recordTypeDirs = fs
+      .readdirSync(adapterPath)
+      .filter((d) => /[a-z]+protocol$/i.test(d))


tbh, I really dont like this flow.

Probably good seperation of code, makes it easier to read, disable only aggregated part, but more efficient to do it in one place instead of writing to a file and then reading it. We can use a flag like generateBreakdown to enable/disable certain code block

can you move the logic here to addProtocolData, i.e compute both the breakdown data & montly/quartly aggregation like how we do chain aggregation there.

then in the file generation state (where buildLabelChartsFromRecords is run atm), we save this pregenerated data as file

g1nt0ki · 2025-09-22T13:08:14Z

defi/src/api2/cron-task/dimensions.ts

      protocol.info = { ...(tvlProtocolInfo ?? {}), };
-      protocol.misc = {      };
-      const infoKeys = ['name', 'defillamaId', 'displayName', 'module', 'category', 'logo', 'chains', 'methodologyURL', 'methodology', 'gecko_id', 'forkedFrom', 'twitter', 'audits', 'description', 'address', 'url', 'audit_links', 'cmcId', 'id', 'github', 'governanceID', 'treasury', 'parentProtocol', 'previousNames', 'hallmarks', 'defaultChartView']
+      protocol.misc = { versionKey: info.versionKey }; // TODO: check, this is not stored in cache correctly and as workaround we are storing it in info object


I have removed versionKey, this should be changed to protocol.misc = {}

g1nt0ki · 2025-09-22T13:12:05Z

defi/src/api2/cron-task/dimensions.ts

+   - dr   = df - dssr, if dr   missing and df & dssr available
+   - dhr  = dr - dssr, if dhr  missing and dr & dssr available
+*/
+function fillDerivedMetrics(proto: AggregatedProtocolOut) {


imo, this refilling should be done when we are pulling data from db and before it gets added to cache, I would add a new field to protocol cache, hasBreakdownData, so if that flag is true or any of the new items have bl field, run this logic to auto add missing fields

…illama-server into feat_breadownCache

0xpeluche added 5 commits September 12, 2025 10:14

wip breakdown

b713975

wip

9a95c46

rm blc add aggregate monthly/quaterly

38bedf7

refacto one file

3d9550b

add routes

660b09a

0xpeluche self-assigned this Sep 15, 2025

g1nt0ki reviewed Sep 15, 2025

View reviewed changes

g1nt0ki requested changes Sep 15, 2025

View reviewed changes

0xpeluche and others added 2 commits September 16, 2025 14:15

some changes, add pro router

c2cf81d

Merge branch 'master' into feat_breadownCache

83151a3

g1nt0ki reviewed Sep 21, 2025

View reviewed changes

g1nt0ki reviewed Sep 22, 2025

View reviewed changes

0xpeluche added 3 commits September 23, 2025 15:03

reply to the various reviews

6259785

Merge branch 'feat_breadownCache' of https://github.com/0xpeluche/def…

0b5ae04

…illama-server into feat_breadownCache

add aggregates

9bbceb1

0xpeluche added 3 commits September 23, 2025 16:07

rm backfill only bl

18ef55c

chore: revert submodule pointers to origin/main

e29d75a

chore: align submodule pointers with upstream/master

9c150ed

0xpeluche force-pushed the feat_breadownCache branch from f6656a7 to 9c150ed Compare October 1, 2025 06:56


		response.metrics = filteredMetrics

		const cleanResponse = JSON.parse(JSON.stringify(response))

		export async function getDimensionBreakdownProtocolFileRoute(req: HyperExpress.Request, res: HyperExpress.Response) {
		const protocolName = req.path_parameters.name?.toLowerCase()

Feat Breakdowns #10613

Are you sure you want to change the base?

Feat Breakdowns #10613

Conversation

0xpeluche commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

g1nt0ki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!