Skip to content

ColBERT 변환 #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

ColBERT 변환 #2

wants to merge 3 commits into from

Conversation

sigridjineth
Copy link
Member

@sigridjineth sigridjineth commented Sep 6, 2025 •
edited
Loading

Position IDs를 attention_mask 기반으로 바꾼 것
→ L0부터 끝까지 수학적 정합성 확보(가장 큰 근본 원인 제거).

PyTorch‑정확 LayerNorm(TorchLayerNormTF1)
→ LN의 미세 오차 제거. embedding 레벨과 모든 블록 잔차경로의 수치 일치 보장.

TF1 그래프 + feed‑assign + 커스텀 Saver(변수 전수집)
→ 그래프에 대형 Const 노드가 남지 않고, 모든 변수가 SavedModel에 포함.
→ TF Serving/Java 호환성과 배포 안정성의 핵심.

Export 단계 Sanity Check(임베딩+LN L0 assert)
→ 잘못된 변환물 저장 자체를 차단하는 마지막 안전장치.

ColBERT head 완전 일치(가중치/바이어스/마스킹/CLS 제외)
→ 실제 검색 품질과 직결되는 헤드가 PT와 수치적으로 동일(MSE=0).

/Users/sigridjineth/Desktop/work/BGE-M3-Model-Converter/.venv/bin/python3.11 /Users/sigridjineth/Desktop/work/BGE-M3-Model-Converter/model_conversion_validator.py 
=== 1) PyTorch ===

[PyTorch] All Layer Outputs:
  Layer 0 hidden state shape: (2, 20, 1024)
  Layer 1 hidden state shape: (2, 20, 1024)
  Layer 2 hidden state shape: (2, 20, 1024)
  Layer 3 hidden state shape: (2, 20, 1024)
  Layer 4 hidden state shape: (2, 20, 1024)
  Layer 5 hidden state shape: (2, 20, 1024)
  Layer 6 hidden state shape: (2, 20, 1024)
  Layer 7 hidden state shape: (2, 20, 1024)
  Layer 8 hidden state shape: (2, 20, 1024)
  Layer 9 hidden state shape: (2, 20, 1024)
  Layer 10 hidden state shape: (2, 20, 1024)
  Layer 11 hidden state shape: (2, 20, 1024)
  Layer 12 hidden state shape: (2, 20, 1024)
  Layer 13 hidden state shape: (2, 20, 1024)
  Layer 14 hidden state shape: (2, 20, 1024)
  Layer 15 hidden state shape: (2, 20, 1024)
  Layer 16 hidden state shape: (2, 20, 1024)
  Layer 17 hidden state shape: (2, 20, 1024)
  Layer 18 hidden state shape: (2, 20, 1024)
  Layer 19 hidden state shape: (2, 20, 1024)
  Layer 20 hidden state shape: (2, 20, 1024)
  Layer 21 hidden state shape: (2, 20, 1024)
  Layer 22 hidden state shape: (2, 20, 1024)
  Layer 23 hidden state shape: (2, 20, 1024)
  Layer 24 hidden state shape: (2, 20, 1024)
=== 2) TensorFlow ===
outputs >> {'colbert_vecs': <tf.Tensor: shape=(2, 19, 1024), dtype=float32, numpy=
array([[[ 0.32928512, -0.1125585 , -0.49183428, ...,  0.16155368,
          0.33396345, -0.00675448],
        [ 0.4307742 ,  0.00873679, -0.2865736 , ...,  0.32808074,
          0.20151775, -0.06308884],
        [ 0.27137956, -0.06170901, -0.25272104, ...,  0.04739508,
          0.41049084,  0.02530823],
        ...,
        [-0.00546088,  0.08859837, -0.3504484 , ...,  0.2625171 ,
          0.46613252,  0.1334329 ],
        [-0.02450071, -0.18421263,  0.03060211, ...,  0.11359002,
          0.48158002, -0.00834481],
        [ 0.4508363 ,  0.19649318, -0.12494306, ...,  0.5596576 ,
          0.07293648, -0.13662192]],

       [[ 0.25521103, -0.00875238, -0.53789777, ...,  0.09722964,
          0.36974305, -0.06560165],
        [ 0.38389987,  0.10751002, -0.37806442, ...,  0.29909363,
          0.22168191, -0.05762064],
        [ 0.16351153,  0.04581645, -0.3656966 , ..., -0.00551193,
          0.43617916, -0.0839855 ],
        ...,
        [ 0.        , -0.        , -0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        , -0.        , -0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        , -0.        , -0.        , ...,  0.        ,
          0.        ,  0.        ]]], dtype=float32)>, 'dense_vecs': <tf.Tensor: shape=(2, 1024), dtype=float32, numpy=
array([[-2.1483502 ,  0.09513041, -1.4056467 , ...,  0.8459837 ,
         0.09479031,  0.11166006],
       [-2.0583854 , -0.19201884, -1.1751288 , ...,  0.7148075 ,
         0.02913851,  0.0878114 ]], dtype=float32)>, 'last_hidden_state': <tf.Tensor: shape=(2, 20, 1024), dtype=float32, numpy=
array([[[-2.1483502 ,  0.09513041, -1.4056467 , ...,  0.8459837 ,
          0.09479031,  0.11166006],
        [-0.56857264,  0.01693045, -0.39471895, ...,  1.0846975 ,
         -0.97512513, -0.695277  ],
        [-0.7711521 ,  0.5267325 , -0.76097715, ...,  0.95354694,
         -0.862591  , -0.38046014],
        ...,
        [-0.90763134, -0.31379372, -0.5904193 , ...,  0.5095541 ,
         -1.1137675 , -0.605915  ],
        [-0.89193636, -0.25980222, -0.31470618, ...,  0.7653711 ,
         -1.3908155 , -0.68685776],
        [-1.0297037 ,  0.96891874,  0.481434  , ...,  1.6668912 ,
         -1.9607738 , -0.8297495 ]],

       [[-2.0583854 , -0.19201884, -1.1751288 , ...,  0.7148075 ,
          0.02913851,  0.0878114 ],
        [-0.57164884, -0.09524751, -0.341056  , ...,  1.016677  ,
         -0.9890139 , -0.59316653],
        [-0.7171066 ,  0.33908647, -0.6335476 , ...,  0.8057392 ,
         -0.8336259 , -0.40080425],
        ...,
        [-0.7006699 ,  0.14596634, -0.18742698, ...,  0.49917614,
         -1.1085962 , -0.22989449],
        [-0.7006699 ,  0.14596634, -0.18742698, ...,  0.49917614,
         -1.1085962 , -0.22989449],
        [-0.7006699 ,  0.14596634, -0.18742698, ...,  0.49917614,
         -1.1085962 , -0.22989449]]], dtype=float32)>, 'hidden_states': <tf.Tensor: shape=(25, 2, 20, 1024), dtype=float32, numpy=
array([[[[-2.01599941e-01, -7.86216557e-03, -2.22602755e-01, ...,
          -4.54514116e-01, -6.05456650e-01, -8.37140083e-01],
         [ 1.85482979e-01,  4.61338162e-01, -5.56154788e-01, ...,
          -5.64155698e-01, -1.91667378e-02,  8.14232647e-01],
         [-3.28781426e-01,  2.24465728e+00,  1.09623373e+00, ...,
           1.83495790e-01, -4.67760861e-01,  8.08617771e-01],
         ...,
         [ 9.67531323e-01, -1.12221932e+00, -1.01219523e+00, ...,
          -3.70522469e-01, -4.62963820e-01, -5.20748258e-01],
         [ 7.82766521e-01,  7.63325572e-01, -3.11636388e-01, ...,
          -4.10208106e-01,  2.65854597e-01,  1.27472147e-01],
         [-3.27237040e-01,  2.76762247e-01, -4.90221024e-01, ...,
           1.82240874e-01, -7.18727410e-02,  8.28074932e-01]],

        [[-2.01599941e-01, -7.86216557e-03, -2.22602755e-01, ...,
          -4.54514116e-01, -6.05456650e-01, -8.37140083e-01],
         [ 1.85482979e-01,  4.61338162e-01, -5.56154788e-01, ...,
          -5.64155698e-01, -1.91667378e-02,  8.14232647e-01],
         [-3.28781426e-01,  2.24465728e+00,  1.09623373e+00, ...,
           1.83495790e-01, -4.67760861e-01,  8.08617771e-01],
         ...,
         [-6.67161345e-02,  9.72707272e-01,  1.32836986e+00, ...,
          -6.10638976e-01,  3.51469517e-01, -3.03919435e-01],
         [-6.67161345e-02,  9.72707272e-01,  1.32836986e+00, ...,
          -6.10638976e-01,  3.51469517e-01, -3.03919435e-01],
         [-6.67161345e-02,  9.72707272e-01,  1.32836986e+00, ...,
          -6.10638976e-01,  3.51469517e-01, -3.03919435e-01]]],


       [[[ 1.15139261e-02, -9.63224471e-03, -8.35065618e-02, ...,
          -1.12153128e-01, -1.41865730e-01, -5.85492104e-02],
         [-5.82076162e-02,  3.15992653e-01, -3.17481995e-01, ...,
          -9.50658694e-02,  2.50744462e-01,  1.58798039e-01],
         [-5.92223883e-01,  2.08463717e+00,  8.75745714e-01, ...,
           7.98759878e-01, -8.08436126e-02,  5.57635188e-01],
         ...,
         [ 1.28773093e+00, -1.15310872e+00, -2.20643282e-01, ...,
           2.33099848e-01, -5.94621658e-01, -5.28985023e-01],
         [ 5.27175188e-01,  2.85304576e-01,  3.95195812e-01, ...,
           3.93138379e-02,  5.01447380e-01,  3.92519198e-02],
         [ 3.22251022e-02,  1.17851168e-01, -1.13468707e-01, ...,
           2.38832712e-01,  7.14738667e-03,  2.67159909e-01]],

        [[ 6.87309355e-03, -6.93711638e-03, -9.27730277e-02, ...,
          -1.19974196e-01, -1.38715670e-01, -5.46134263e-02],
         [-1.32532001e-01,  3.25333476e-01, -3.09728384e-01, ...,
          -1.20390207e-01,  2.98781246e-01,  1.87307030e-01],
         [-5.61740041e-01,  2.10902214e+00,  8.78870845e-01, ...,
           6.95444345e-01, -2.88398862e-02,  5.09737253e-01],
         ...,
         [-2.58515626e-01,  7.34064162e-01,  1.23776570e-01, ...,
          -2.25045905e-02,  4.01488125e-01,  1.49843216e-01],
         [-2.58515626e-01,  7.34064162e-01,  1.23776570e-01, ...,
          -2.25045905e-02,  4.01488125e-01,  1.49843216e-01],
         [-2.58515626e-01,  7.34064162e-01,  1.23776570e-01, ...,
          -2.25045905e-02,  4.01488125e-01,  1.49843216e-01]]],


       [[[ 1.34169608e-02,  5.30507565e-02,  3.60805616e-02, ...,
          -9.74385515e-02, -1.07595146e-01, -6.01698682e-02],
         [ 1.18990012e-01, -5.36777973e-02, -6.22657537e-01, ...,
           3.68920952e-01,  4.24076498e-01,  2.94018388e-01],
         [-5.32341897e-01,  1.70937681e+00,  5.65685630e-01, ...,
           9.62958694e-01, -1.60964429e-01,  5.49282849e-01],
         ...,
         [ 6.78164840e-01, -1.19416153e+00, -5.33385754e-01, ...,
           1.70671880e-01, -3.58455956e-01, -1.57322153e-01],
         [ 3.71268570e-01, -2.04027861e-01,  2.67829478e-01, ...,
           2.43594289e-01,  5.82521975e-01, -7.08120987e-02],
         [-3.57711092e-02,  1.61767900e-02, -1.43706620e-01, ...,
          -1.23594627e-02, -3.57559174e-02,  2.62436401e-02]],

        [[ 1.60720460e-02,  5.71908355e-02,  4.41372693e-02, ...,
          -9.28789079e-02, -1.18164346e-01, -5.35207838e-02],
         [ 6.57682121e-02,  1.25494599e-03, -6.05309725e-01, ...,
           3.53526890e-01,  4.27301109e-01,  3.21716428e-01],
         [-4.74866033e-01,  1.72289586e+00,  5.67326069e-01, ...,
           9.14279461e-01, -1.18581727e-01,  5.56868494e-01],
         ...,
         [-2.30614603e-01,  3.87703001e-01, -4.50892746e-01, ...,
           5.78549802e-02,  4.29446697e-01, -3.20436329e-01],
         [-2.30614603e-01,  3.87703001e-01, -4.50892746e-01, ...,
           5.78549802e-02,  4.29446697e-01, -3.20436329e-01],
         [-2.30614603e-01,  3.87703001e-01, -4.50892746e-01, ...,
           5.78549802e-02,  4.29446697e-01, -3.20436329e-01]]],


       ...,


       [[[-1.27540874e+00, -4.58678557e-03, -1.15098011e+00, ...,
           2.75576651e-01,  4.24775839e-01, -1.34751365e-01],
         [-2.49396294e-01, -6.80620894e-02, -1.71841070e-01, ...,
           4.79258776e-01,  1.40869662e-01, -1.39913023e-01],
         [-6.98646545e-01,  3.10836583e-01, -3.84247184e-01, ...,
           1.26158655e-01, -1.88892350e-01, -3.77888262e-01],
         ...,
         [-6.14213347e-01, -4.17930275e-01, -4.63818282e-01, ...,
          -1.97953716e-01, -2.53939152e-01, -2.58642375e-01],
         [-5.84038913e-01, -3.64670962e-01, -1.53524932e-02, ...,
           3.49923447e-02, -2.30085582e-01, -2.65964717e-01],
         [ 4.45536971e-02,  2.25473363e-02,  3.39261629e-03, ...,
           2.04134006e-02, -3.53158340e-02,  2.05262657e-03]],

        [[-1.17254758e+00, -9.58653912e-02, -1.03250802e+00, ...,
           2.39492059e-01,  3.70547652e-01, -7.36531839e-02],
         [-2.82854468e-01, -4.78258729e-02, -6.67395666e-02, ...,
           3.64551961e-01,  5.01017421e-02, -7.16470256e-02],
         [-6.40267134e-01,  2.79448450e-01, -3.05311650e-01, ...,
           4.36218381e-02, -1.28343105e-01, -3.89768004e-01],
         ...,
         [-3.03478748e-01,  9.51443017e-02, -5.02063930e-02, ...,
           1.23201013e-01, -6.20901883e-01, -3.19725633e-01],
         [-3.03478748e-01,  9.51443017e-02, -5.02063930e-02, ...,
           1.23201013e-01, -6.20901883e-01, -3.19725633e-01],
         [-3.03478748e-01,  9.51443017e-02, -5.02063930e-02, ...,
           1.23201013e-01, -6.20901883e-01, -3.19725633e-01]]],


       [[[-2.00921988e+00, -1.21369381e-02, -1.64234626e+00, ...,
           2.14234337e-01,  7.40045905e-01,  1.20419815e-01],
         [-1.17290273e-01, -8.03189352e-02, -3.10231715e-01, ...,
           1.90606087e-01, -3.75957824e-02, -3.02683175e-01],
         [-6.84996724e-01,  1.08053543e-01, -4.60588634e-01, ...,
           9.46937278e-02, -1.51001900e-01, -2.41469443e-01],
         ...,
         [-4.42221254e-01, -3.83497745e-01, -4.23948556e-01, ...,
          -1.74611241e-01, -1.94339588e-01, -2.54128247e-01],
         [-3.47865373e-01, -2.32349113e-01, -1.63565040e-01, ...,
           1.15092471e-02, -3.56334984e-01, -2.86574811e-01],
         [ 1.51067227e-03,  1.55011695e-02, -8.05441942e-03, ...,
           3.38953808e-02, -1.27834594e-02,  1.82832778e-02]],

        [[-1.93117547e+00, -2.39095777e-01, -1.38717842e+00, ...,
           1.16140895e-01,  6.38662398e-01,  9.80680883e-02],
         [-2.10894227e-01, -1.14054471e-01, -1.95792288e-01, ...,
           1.04203083e-01, -5.36815673e-02, -2.12688759e-01],
         [-6.32562876e-01,  6.19919859e-02, -3.61026317e-01, ...,
          -6.83122315e-03, -1.35096952e-01, -2.41264805e-01],
         ...,
         [-4.97520030e-01, -1.06647797e-01, -8.58810171e-02, ...,
          -4.70564067e-02, -4.36508358e-01, -2.52092451e-01],
         [-4.97520030e-01, -1.06647797e-01, -8.58810171e-02, ...,
          -4.70564067e-02, -4.36508358e-01, -2.52092451e-01],
         [-4.97520030e-01, -1.06647797e-01, -8.58810171e-02, ...,
          -4.70564067e-02, -4.36508358e-01, -2.52092451e-01]]],


       [[[-2.14835024e+00,  9.51304063e-02, -1.40564668e+00, ...,
           8.45983684e-01,  9.47903097e-02,  1.11660063e-01],
         [-5.68572640e-01,  1.69304498e-02, -3.94718945e-01, ...,
           1.08469748e+00, -9.75125134e-01, -6.95276976e-01],
         [-7.71152079e-01,  5.26732504e-01, -7.60977149e-01, ...,
           9.53546941e-01, -8.62591028e-01, -3.80460143e-01],
         ...,
         [-9.07631338e-01, -3.13793719e-01, -5.90419292e-01, ...,
           5.09554088e-01, -1.11376750e+00, -6.05915010e-01],
         [-8.91936362e-01, -2.59802222e-01, -3.14706177e-01, ...,
           7.65371084e-01, -1.39081550e+00, -6.86857760e-01],
         [-1.02970374e+00,  9.68918741e-01,  4.81433988e-01, ...,
           1.66689122e+00, -1.96077383e+00, -8.29749525e-01]],

        [[-2.05838537e+00, -1.92018837e-01, -1.17512882e+00, ...,
           7.14807510e-01,  2.91385055e-02,  8.78113955e-02],
         [-5.71648836e-01, -9.52475071e-02, -3.41055989e-01, ...,
           1.01667702e+00, -9.89013910e-01, -5.93166530e-01],
         [-7.17106581e-01,  3.39086473e-01, -6.33547604e-01, ...,
           8.05739224e-01, -8.33625913e-01, -4.00804251e-01],
         ...,
         [-7.00669885e-01,  1.45966336e-01, -1.87426984e-01, ...,
           4.99176145e-01, -1.10859621e+00, -2.29894489e-01],
         [-7.00669885e-01,  1.45966336e-01, -1.87426984e-01, ...,
           4.99176145e-01, -1.10859621e+00, -2.29894489e-01],
         [-7.00669885e-01,  1.45966336e-01, -1.87426984e-01, ...,
           4.99176145e-01, -1.10859621e+00, -2.29894489e-01]]]],
      dtype=float32)>}
hiddens, [[[[-2.01599941e-01 -7.86216557e-03 -2.22602755e-01 ... -4.54514116e-01
    -6.05456650e-01 -8.37140083e-01]
   [ 1.85482979e-01  4.61338162e-01 -5.56154788e-01 ... -5.64155698e-01
    -1.91667378e-02  8.14232647e-01]
   [-3.28781426e-01  2.24465728e+00  1.09623373e+00 ...  1.83495790e-01
    -4.67760861e-01  8.08617771e-01]
   ...
   [ 9.67531323e-01 -1.12221932e+00 -1.01219523e+00 ... -3.70522469e-01
    -4.62963820e-01 -5.20748258e-01]
   [ 7.82766521e-01  7.63325572e-01 -3.11636388e-01 ... -4.10208106e-01
     2.65854597e-01  1.27472147e-01]
   [-3.27237040e-01  2.76762247e-01 -4.90221024e-01 ...  1.82240874e-01
    -7.18727410e-02  8.28074932e-01]]

  [[-2.01599941e-01 -7.86216557e-03 -2.22602755e-01 ... -4.54514116e-01
    -6.05456650e-01 -8.37140083e-01]
   [ 1.85482979e-01  4.61338162e-01 -5.56154788e-01 ... -5.64155698e-01
    -1.91667378e-02  8.14232647e-01]
   [-3.28781426e-01  2.24465728e+00  1.09623373e+00 ...  1.83495790e-01
    -4.67760861e-01  8.08617771e-01]
   ...
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]]]


 [[[ 1.15139261e-02 -9.63224471e-03 -8.35065618e-02 ... -1.12153128e-01
    -1.41865730e-01 -5.85492104e-02]
   [-5.82076162e-02  3.15992653e-01 -3.17481995e-01 ... -9.50658694e-02
     2.50744462e-01  1.58798039e-01]
   [-5.92223883e-01  2.08463717e+00  8.75745714e-01 ...  7.98759878e-01
    -8.08436126e-02  5.57635188e-01]
   ...
   [ 1.28773093e+00 -1.15310872e+00 -2.20643282e-01 ...  2.33099848e-01
    -5.94621658e-01 -5.28985023e-01]
   [ 5.27175188e-01  2.85304576e-01  3.95195812e-01 ...  3.93138379e-02
     5.01447380e-01  3.92519198e-02]
   [ 3.22251022e-02  1.17851168e-01 -1.13468707e-01 ...  2.38832712e-01
     7.14738667e-03  2.67159909e-01]]

  [[ 6.87309355e-03 -6.93711638e-03 -9.27730277e-02 ... -1.19974196e-01
    -1.38715670e-01 -5.46134263e-02]
   [-1.32532001e-01  3.25333476e-01 -3.09728384e-01 ... -1.20390207e-01
     2.98781246e-01  1.87307030e-01]
   [-5.61740041e-01  2.10902214e+00  8.78870845e-01 ...  6.95444345e-01
    -2.88398862e-02  5.09737253e-01]
   ...
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]]]


 [[[ 1.34169608e-02  5.30507565e-02  3.60805616e-02 ... -9.74385515e-02
    -1.07595146e-01 -6.01698682e-02]
   [ 1.18990012e-01 -5.36777973e-02 -6.22657537e-01 ...  3.68920952e-01
     4.24076498e-01  2.94018388e-01]
   [-5.32341897e-01  1.70937681e+00  5.65685630e-01 ...  9.62958694e-01
    -1.60964429e-01  5.49282849e-01]
   ...
   [ 6.78164840e-01 -1.19416153e+00 -5.33385754e-01 ...  1.70671880e-01
    -3.58455956e-01 -1.57322153e-01]
   [ 3.71268570e-01 -2.04027861e-01  2.67829478e-01 ...  2.43594289e-01
     5.82521975e-01 -7.08120987e-02]
   [-3.57711092e-02  1.61767900e-02 -1.43706620e-01 ... -1.23594627e-02
    -3.57559174e-02  2.62436401e-02]]

  [[ 1.60720460e-02  5.71908355e-02  4.41372693e-02 ... -9.28789079e-02
    -1.18164346e-01 -5.35207838e-02]
   [ 6.57682121e-02  1.25494599e-03 -6.05309725e-01 ...  3.53526890e-01
     4.27301109e-01  3.21716428e-01]
   [-4.74866033e-01  1.72289586e+00  5.67326069e-01 ...  9.14279461e-01
    -1.18581727e-01  5.56868494e-01]
   ...
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]]]


 ...


 [[[-1.27540874e+00 -4.58678557e-03 -1.15098011e+00 ...  2.75576651e-01
     4.24775839e-01 -1.34751365e-01]
   [-2.49396294e-01 -6.80620894e-02 -1.71841070e-01 ...  4.79258776e-01
     1.40869662e-01 -1.39913023e-01]
   [-6.98646545e-01  3.10836583e-01 -3.84247184e-01 ...  1.26158655e-01
    -1.88892350e-01 -3.77888262e-01]
   ...
   [-6.14213347e-01 -4.17930275e-01 -4.63818282e-01 ... -1.97953716e-01
    -2.53939152e-01 -2.58642375e-01]
   [-5.84038913e-01 -3.64670962e-01 -1.53524932e-02 ...  3.49923447e-02
    -2.30085582e-01 -2.65964717e-01]
   [ 4.45536971e-02  2.25473363e-02  3.39261629e-03 ...  2.04134006e-02
    -3.53158340e-02  2.05262657e-03]]

  [[-1.17254758e+00 -9.58653912e-02 -1.03250802e+00 ...  2.39492059e-01
     3.70547652e-01 -7.36531839e-02]
   [-2.82854468e-01 -4.78258729e-02 -6.67395666e-02 ...  3.64551961e-01
     5.01017421e-02 -7.16470256e-02]
   [-6.40267134e-01  2.79448450e-01 -3.05311650e-01 ...  4.36218381e-02
    -1.28343105e-01 -3.89768004e-01]
   ...
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]]]


 [[[-2.00921988e+00 -1.21369381e-02 -1.64234626e+00 ...  2.14234337e-01
     7.40045905e-01  1.20419815e-01]
   [-1.17290273e-01 -8.03189352e-02 -3.10231715e-01 ...  1.90606087e-01
    -3.75957824e-02 -3.02683175e-01]
   [-6.84996724e-01  1.08053543e-01 -4.60588634e-01 ...  9.46937278e-02
    -1.51001900e-01 -2.41469443e-01]
   ...
   [-4.42221254e-01 -3.83497745e-01 -4.23948556e-01 ... -1.74611241e-01
    -1.94339588e-01 -2.54128247e-01]
   [-3.47865373e-01 -2.32349113e-01 -1.63565040e-01 ...  1.15092471e-02
    -3.56334984e-01 -2.86574811e-01]
   [ 1.51067227e-03  1.55011695e-02 -8.05441942e-03 ...  3.38953808e-02
    -1.27834594e-02  1.82832778e-02]]

  [[-1.93117547e+00 -2.39095777e-01 -1.38717842e+00 ...  1.16140895e-01
     6.38662398e-01  9.80680883e-02]
   [-2.10894227e-01 -1.14054471e-01 -1.95792288e-01 ...  1.04203083e-01
    -5.36815673e-02 -2.12688759e-01]
   [-6.32562876e-01  6.19919859e-02 -3.61026317e-01 ... -6.83122315e-03
    -1.35096952e-01 -2.41264805e-01]
   ...
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]]]


 [[[-2.14835024e+00  9.51304063e-02 -1.40564668e+00 ...  8.45983684e-01
     9.47903097e-02  1.11660063e-01]
   [-5.68572640e-01  1.69304498e-02 -3.94718945e-01 ...  1.08469748e+00
    -9.75125134e-01 -6.95276976e-01]
   [-7.71152079e-01  5.26732504e-01 -7.60977149e-01 ...  9.53546941e-01
    -8.62591028e-01 -3.80460143e-01]
   ...
   [-9.07631338e-01 -3.13793719e-01 -5.90419292e-01 ...  5.09554088e-01
    -1.11376750e+00 -6.05915010e-01]
   [-8.91936362e-01 -2.59802222e-01 -3.14706177e-01 ...  7.65371084e-01
    -1.39081550e+00 -6.86857760e-01]
   [-1.02970374e+00  9.68918741e-01  4.81433988e-01 ...  1.66689122e+00
    -1.96077383e+00 -8.29749525e-01]]

  [[-2.05838537e+00 -1.92018837e-01 -1.17512882e+00 ...  7.14807510e-01
     2.91385055e-02  8.78113955e-02]
   [-5.71648836e-01 -9.52475071e-02 -3.41055989e-01 ...  1.01667702e+00
    -9.89013910e-01 -5.93166530e-01]
   [-7.17106581e-01  3.39086473e-01 -6.33547604e-01 ...  8.05739224e-01
    -8.33625913e-01 -4.00804251e-01]
   ...
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]]]]
L0 CLS head(PT)[:8]: [-0.20159993 -0.00786221 -0.22260273 -0.30351293  0.7716978  -0.12234226
  0.00731593  0.07493226]
L0 CLS head(TF)[:8]: [-0.20159994 -0.00786217 -0.22260275 -0.30351293  0.7716979  -0.12234224
  0.0073159   0.07493228]

=== 3) Compare ===
PT shape: (2, 1024) TF shape: (2, 1024)
Cosine: ['1.0000', '1.0000']
MSE: 2.2349812972555227e-12
tf_layers, [[[[-2.01599941e-01 -7.86216557e-03 -2.22602755e-01 ... -4.54514116e-01
    -6.05456650e-01 -8.37140083e-01]
   [ 1.85482979e-01  4.61338162e-01 -5.56154788e-01 ... -5.64155698e-01
    -1.91667378e-02  8.14232647e-01]
   [-3.28781426e-01  2.24465728e+00  1.09623373e+00 ...  1.83495790e-01
    -4.67760861e-01  8.08617771e-01]
   ...
   [ 9.67531323e-01 -1.12221932e+00 -1.01219523e+00 ... -3.70522469e-01
    -4.62963820e-01 -5.20748258e-01]
   [ 7.82766521e-01  7.63325572e-01 -3.11636388e-01 ... -4.10208106e-01
     2.65854597e-01  1.27472147e-01]
   [-3.27237040e-01  2.76762247e-01 -4.90221024e-01 ...  1.82240874e-01
    -7.18727410e-02  8.28074932e-01]]

  [[-2.01599941e-01 -7.86216557e-03 -2.22602755e-01 ... -4.54514116e-01
    -6.05456650e-01 -8.37140083e-01]
   [ 1.85482979e-01  4.61338162e-01 -5.56154788e-01 ... -5.64155698e-01
    -1.91667378e-02  8.14232647e-01]
   [-3.28781426e-01  2.24465728e+00  1.09623373e+00 ...  1.83495790e-01
    -4.67760861e-01  8.08617771e-01]
   ...
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]
   [-6.67161345e-02  9.72707272e-01  1.32836986e+00 ... -6.10638976e-01
     3.51469517e-01 -3.03919435e-01]]]


 [[[ 1.15139261e-02 -9.63224471e-03 -8.35065618e-02 ... -1.12153128e-01
    -1.41865730e-01 -5.85492104e-02]
   [-5.82076162e-02  3.15992653e-01 -3.17481995e-01 ... -9.50658694e-02
     2.50744462e-01  1.58798039e-01]
   [-5.92223883e-01  2.08463717e+00  8.75745714e-01 ...  7.98759878e-01
    -8.08436126e-02  5.57635188e-01]
   ...
   [ 1.28773093e+00 -1.15310872e+00 -2.20643282e-01 ...  2.33099848e-01
    -5.94621658e-01 -5.28985023e-01]
   [ 5.27175188e-01  2.85304576e-01  3.95195812e-01 ...  3.93138379e-02
     5.01447380e-01  3.92519198e-02]
   [ 3.22251022e-02  1.17851168e-01 -1.13468707e-01 ...  2.38832712e-01
     7.14738667e-03  2.67159909e-01]]

  [[ 6.87309355e-03 -6.93711638e-03 -9.27730277e-02 ... -1.19974196e-01
    -1.38715670e-01 -5.46134263e-02]
   [-1.32532001e-01  3.25333476e-01 -3.09728384e-01 ... -1.20390207e-01
     2.98781246e-01  1.87307030e-01]
   [-5.61740041e-01  2.10902214e+00  8.78870845e-01 ...  6.95444345e-01
    -2.88398862e-02  5.09737253e-01]
   ...
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]
   [-2.58515626e-01  7.34064162e-01  1.23776570e-01 ... -2.25045905e-02
     4.01488125e-01  1.49843216e-01]]]


 [[[ 1.34169608e-02  5.30507565e-02  3.60805616e-02 ... -9.74385515e-02
    -1.07595146e-01 -6.01698682e-02]
   [ 1.18990012e-01 -5.36777973e-02 -6.22657537e-01 ...  3.68920952e-01
     4.24076498e-01  2.94018388e-01]
   [-5.32341897e-01  1.70937681e+00  5.65685630e-01 ...  9.62958694e-01
    -1.60964429e-01  5.49282849e-01]
   ...
   [ 6.78164840e-01 -1.19416153e+00 -5.33385754e-01 ...  1.70671880e-01
    -3.58455956e-01 -1.57322153e-01]
   [ 3.71268570e-01 -2.04027861e-01  2.67829478e-01 ...  2.43594289e-01
     5.82521975e-01 -7.08120987e-02]
   [-3.57711092e-02  1.61767900e-02 -1.43706620e-01 ... -1.23594627e-02
    -3.57559174e-02  2.62436401e-02]]

  [[ 1.60720460e-02  5.71908355e-02  4.41372693e-02 ... -9.28789079e-02
    -1.18164346e-01 -5.35207838e-02]
   [ 6.57682121e-02  1.25494599e-03 -6.05309725e-01 ...  3.53526890e-01
     4.27301109e-01  3.21716428e-01]
   [-4.74866033e-01  1.72289586e+00  5.67326069e-01 ...  9.14279461e-01
    -1.18581727e-01  5.56868494e-01]
   ...
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]
   [-2.30614603e-01  3.87703001e-01 -4.50892746e-01 ...  5.78549802e-02
     4.29446697e-01 -3.20436329e-01]]]


 ...


 [[[-1.27540874e+00 -4.58678557e-03 -1.15098011e+00 ...  2.75576651e-01
     4.24775839e-01 -1.34751365e-01]
   [-2.49396294e-01 -6.80620894e-02 -1.71841070e-01 ...  4.79258776e-01
     1.40869662e-01 -1.39913023e-01]
   [-6.98646545e-01  3.10836583e-01 -3.84247184e-01 ...  1.26158655e-01
    -1.88892350e-01 -3.77888262e-01]
   ...
   [-6.14213347e-01 -4.17930275e-01 -4.63818282e-01 ... -1.97953716e-01
    -2.53939152e-01 -2.58642375e-01]
   [-5.84038913e-01 -3.64670962e-01 -1.53524932e-02 ...  3.49923447e-02
    -2.30085582e-01 -2.65964717e-01]
   [ 4.45536971e-02  2.25473363e-02  3.39261629e-03 ...  2.04134006e-02
    -3.53158340e-02  2.05262657e-03]]

  [[-1.17254758e+00 -9.58653912e-02 -1.03250802e+00 ...  2.39492059e-01
     3.70547652e-01 -7.36531839e-02]
   [-2.82854468e-01 -4.78258729e-02 -6.67395666e-02 ...  3.64551961e-01
     5.01017421e-02 -7.16470256e-02]
   [-6.40267134e-01  2.79448450e-01 -3.05311650e-01 ...  4.36218381e-02
    -1.28343105e-01 -3.89768004e-01]
   ...
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]
   [-3.03478748e-01  9.51443017e-02 -5.02063930e-02 ...  1.23201013e-01
    -6.20901883e-01 -3.19725633e-01]]]


 [[[-2.00921988e+00 -1.21369381e-02 -1.64234626e+00 ...  2.14234337e-01
     7.40045905e-01  1.20419815e-01]
   [-1.17290273e-01 -8.03189352e-02 -3.10231715e-01 ...  1.90606087e-01
    -3.75957824e-02 -3.02683175e-01]
   [-6.84996724e-01  1.08053543e-01 -4.60588634e-01 ...  9.46937278e-02
    -1.51001900e-01 -2.41469443e-01]
   ...
   [-4.42221254e-01 -3.83497745e-01 -4.23948556e-01 ... -1.74611241e-01
    -1.94339588e-01 -2.54128247e-01]
   [-3.47865373e-01 -2.32349113e-01 -1.63565040e-01 ...  1.15092471e-02
    -3.56334984e-01 -2.86574811e-01]
   [ 1.51067227e-03  1.55011695e-02 -8.05441942e-03 ...  3.38953808e-02
    -1.27834594e-02  1.82832778e-02]]

  [[-1.93117547e+00 -2.39095777e-01 -1.38717842e+00 ...  1.16140895e-01
     6.38662398e-01  9.80680883e-02]
   [-2.10894227e-01 -1.14054471e-01 -1.95792288e-01 ...  1.04203083e-01
    -5.36815673e-02 -2.12688759e-01]
   [-6.32562876e-01  6.19919859e-02 -3.61026317e-01 ... -6.83122315e-03
    -1.35096952e-01 -2.41264805e-01]
   ...
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]
   [-4.97520030e-01 -1.06647797e-01 -8.58810171e-02 ... -4.70564067e-02
    -4.36508358e-01 -2.52092451e-01]]]


 [[[-2.14835024e+00  9.51304063e-02 -1.40564668e+00 ...  8.45983684e-01
     9.47903097e-02  1.11660063e-01]
   [-5.68572640e-01  1.69304498e-02 -3.94718945e-01 ...  1.08469748e+00
    -9.75125134e-01 -6.95276976e-01]
   [-7.71152079e-01  5.26732504e-01 -7.60977149e-01 ...  9.53546941e-01
    -8.62591028e-01 -3.80460143e-01]
   ...
   [-9.07631338e-01 -3.13793719e-01 -5.90419292e-01 ...  5.09554088e-01
    -1.11376750e+00 -6.05915010e-01]
   [-8.91936362e-01 -2.59802222e-01 -3.14706177e-01 ...  7.65371084e-01
    -1.39081550e+00 -6.86857760e-01]
   [-1.02970374e+00  9.68918741e-01  4.81433988e-01 ...  1.66689122e+00
    -1.96077383e+00 -8.29749525e-01]]

  [[-2.05838537e+00 -1.92018837e-01 -1.17512882e+00 ...  7.14807510e-01
     2.91385055e-02  8.78113955e-02]
   [-5.71648836e-01 -9.52475071e-02 -3.41055989e-01 ...  1.01667702e+00
    -9.89013910e-01 -5.93166530e-01]
   [-7.17106581e-01  3.39086473e-01 -6.33547604e-01 ...  8.05739224e-01
    -8.33625913e-01 -4.00804251e-01]
   ...
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]
   [-7.00669885e-01  1.45966336e-01 -1.87426984e-01 ...  4.99176145e-01
    -1.10859621e+00 -2.29894489e-01]]]]

[Layer-wise] Cosine (PT vs TF):
  Layer 00  cos=1.0000  mse=0.000000
  Layer 01  cos=1.0000  mse=0.000000
  Layer 02  cos=1.0000  mse=0.000000
  Layer 03  cos=1.0000  mse=0.000000
  Layer 04  cos=1.0000  mse=0.000000
  Layer 05  cos=1.0000  mse=0.000000
  Layer 06  cos=1.0000  mse=0.000000
  Layer 07  cos=1.0000  mse=0.000000
  Layer 08  cos=1.0000  mse=0.000000
  Layer 09  cos=1.0000  mse=0.000000
  Layer 10  cos=1.0000  mse=0.000000
  Layer 11  cos=1.0000  mse=0.000000
  Layer 12  cos=1.0000  mse=0.000000
  Layer 13  cos=1.0000  mse=0.000000
  Layer 14  cos=1.0000  mse=0.000000
  Layer 15  cos=1.0000  mse=0.000000
  Layer 16  cos=1.0000  mse=0.000000
  Layer 17  cos=1.0000  mse=0.000000
  Layer 18  cos=1.0000  mse=0.000000
  Layer 19  cos=1.0000  mse=0.000000
  Layer 20  cos=1.0000  mse=0.000000
  Layer 21  cos=1.0000  mse=0.000000
  Layer 22  cos=1.0000  mse=0.000000
  Layer 23  cos=1.0000  mse=0.000000
  Layer 24  cos=1.0000  mse=0.000000
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Manual vs PT  MSE: 4.7380177e-15
Manual vs TF  MSE: 4.9299562e-15
Fetching 30 files: 100%|██████████| 30/30 [00:00<00:00, 93137.76it/s]
/Users/sigridjineth/Desktop/work/BGE-M3-Model-Converter/model_conversion_validator.py:121: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  st = torch.load(p, map_location="cpu")

[ColBERT(valid)] mse=0.00000000  cos=1.000000
[ColBERT(all-pos)] mse=0.00000000  cos=0.763158
[last_hidden(valid)] mse=0.00000000  cos=1.000000
[determinism] max_abs_diff: 0.0

Process finished with exit code 0

@sigridjineth sigridjineth changed the title feat: colbert_vecs ColBERT 변환 Sep 7, 2025
@sigridjineth sigridjineth marked this pull request as ready for review September 7, 2025 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@sigridjineth