g PddlZddlTddlZddlZdZdZedfdZ d dZ d!dZ e Z d d Z d ZeZd"d Zd ZdZeZd#dZeZd#dZGddZd$dZGddZedfdZd%dZGddZdZeZdZe Z!d&dZ dS)'N)*ctj|d}|j\}}tj||fd}tj||fd}t j}t ||_t ||_||_ ||_ | | |t || ||fS)zPreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDhas R/var/www/html/mpstechhub/venv/lib/python3.11/site-packages/faiss/extra_wrappers.pykminr  i 8 8 8E ;DAq !Qw'''A !Qy)))A  $ & &B a[[BF a[[BF BE BDJJLLLGGAxJJLLL a4Kctj|d}|j\}}tj||fd}tj||fd}t j}t ||_t ||_||_ ||_ | | |t || ||fS)zOreturn k largest values (and their indices) of the lines of a float32 arrayrrr)r r r r r float_minheap_array_trrrrrrrrrs rkmaxr$+r r!c tj|d}tj|d}|j\}}|j\}}||ksJtj||fd}|tkrVs5 (1I & & &Cx}}ch--- Jr!ctj|d}|$tt||j|n$t t||j|||SNrr)r r& int64_randrr:int64_rand_max)rr<vmaxr=s rrandintrD\sY (1G $ $ $C |8C==#(D1111x}}chd;;; Jr!cxtj|d}tt||j||Sr8)r r& float_randnrr:r;s rrandnrGhs5 (1I & & &C sx... Jr!c"|d}|jdkr"t|jt |S|j\}}t j|d}t||t |t ||S)z> compute a checksum for quick-and-dirty comparisons of arrays uint8uint64r) viewndim bvec_checksumr:rr r r bvecs_checksum)arr1css rchecksumrRnsx wAv{{QVXa[[111 7DAq !8 $ $ $B1a!hrll333 Ir!cttj||fd}t||t|||Sr8)r r&rand_smooth_vectors_cr)rr1r<r=s rrand_smooth_vectorsrVzs9 (Aq6 + + +C!Q t444 Jr!c ptj|d}tj|d}|jd}|jd|ksJ|jd|jd}}d}t|D]=}|t |t |||t ||z }>|S)z< size of intersection between each line of two result tablesrrrrJ)r r r rangeranklist_intersection_sizer)I1I2rk1k2ninteris reval_intersectionr`s b 0 0 0B b 0 0 0B  A 8A;!     Xa["(1+B F 1XX66, AXbe__66 6 Mr!cnt|jd|jdt|dS)NrJr)fvec_renorm_L2r rxs r normalize_L2res,171:qwqz8A;;77777r!c tj|d}|$t|dz}tj|dzd}tj|jd}t |jtj| d|tj|tj||||fS)aPerform a bucket sort on a table of integers. Parameters ---------- tab : array_like elements to sort, max value nbucket - 1 nbucket : integer number of buckets, None if unknown nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) perm : array_like perm[lims[i] : lims[i + 1]] contains the indices of bucket #i (size tab.size) rrNrJrK) r r intmaxr&r: bucket_sort_cr rrL)tabnbucketntlimsperms r bucket_sortros& s' 2 2 2Ccggii!m$$ 8GaKw / / /D 8CHG , , ,D %.(!3!344%%u~d';';  :r!c 8|jdks |jdksJ|j\}}|$t|dz}t j|dzd}t ||tj||tj|||S)aPerform a bucket sort on a matrix, recording the original row of each element. Parameters ---------- tab : array_like array of size (N, ncol) that contains the bucket ids, maximum value nbucket - 1. On output, it the elements are shuffled such that the flat array tab.ravel()[lims[i] : lims[i + 1]] contains the row numbers of each bucket entry. nbucket : integer number of buckets (the maximum value in tab should be nbucket - 1) nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) int32rNrJr) rr rgrhr r&matrix_bucket_sort_inplace_cr r)rjrkrlnrowncolrms rmatrix_bucket_sort_inplacerus, 9  39#7#7#7#7JD$cggii!m$$ 8GaKw / / /D  dEN3''%%  Kr!c,eZdZdZddZdZdZdZdS) ResultHeapz_Accumulate query results from a sliced dataset. The final result will be in self.D, self.I.Fctj||fd|_tj||fd|_||c|_|_|rt }nt}||_||_t|j|_ t|j|_ | ||_ dS)z nq: number of query vectors, k: number of results per query keep_max: keep the top-k maximum values instead of the minima rrrN)r r rrr0rr#rrrrrrheaps)selfr0rkeep_maxrys r__init__zResultHeap.__init__s 2q'1112q'333a  ,)++EE)++ETV$$ TV$$   r!c|j\}}tj|d}tj|d}|j||fksJ||jksJ|j|t |t ||dS)z Add results for all heaps D, I should be of size (nh, nres) D, I do not need to be in a particular order (heap or sorted) rrrN)r r r r0ry addn_with_idsr)rzrrr0kds r add_resultzResultHeap.add_results B  ) 4 4 4  ' 2 2 2w2r(""""TW}}}}    QKK     r!c |j\}}|t|ksJ|jdkr|j|jks|jdkr |j|fksJtj|d}tj|d}tj|d}|jdkrdn|}|j|t||t|t||dS)z Add results for a subset of heaps. D, I should hold resutls for all the subset as a special case, if I is 1D, then all ids are assumed to be the same rJrrrrN)r lenrMr r ryaddn_query_subset_with_idsr)rzsubsetrrnsubsetr id_strides radd_result_subsetzResultHeap.add_result_subsets g #f++%%%% FaKKAGqw.. FaKKAGv----  ) 4 4 4  ' 2 2 2%fG<<<1AA"  -- Xf%%  Xa[[)     r!c8|jdSN)ryr)rzs rfinalizezResultHeap.finalizes r!NF)__name__ __module__ __qualname____doc__r|rrrr!rrwrws_(      *r!rwFc b|j|jksJ|j\}}}tj||f|j}tj||f|j}|rtnt }||||t |t |t |t |||fS)z Merge a set of sorted knn-results obtained from different shards in a dataset Dall and Iall are of size (nshard, nq, k) each D[i, j] should be sorted returns D, I of size (nq, k) as the merged result set r)r r r&rmerge_knn_results_CMaxmerge_knn_results_CMinr) DallIallr{nshardrrDnewInewfuncs rmerge_knn_resultsrs : # # # #:LFAq 8QF$* - - -D 8QF$* - - -D%- I ! !3IDD 1f :r!c eZdZdZdZdZdS)MapInt64ToInt64c"ttj||_|d|jzks Jd||_tj|dfd|_tj|jt|jdS)Nrzneed power of 2 capacityrr) rgr log2 log2_capacitycapacityr&rjr hashtable_int64_to_int64_initr)rzrs rr|zMapInt64ToInt64.__init__3s !2!2331 222224N222  8XqM999 +D,>@R@RSSSSSr!c |j\}|j|fksJtj|jt |j|t |t |dSr)r r hashtable_int64_to_int64_addrrrj)rzkeysvalsrs raddzMapInt64ToInt64.add:sd ZzaT!!!! *   2 2 x~~x~~ / / / / /r!c |j\}tj|fd}tj|jt |j|t |t ||Sr@)r r r&r hashtable_int64_to_int64_lookuprrrj)rzrrrs rlookupzMapInt64ToInt64.lookupAsb ZxG,,, -   2 2 x~~x~~ / / / r!N)rrrr|rrrr!rrr1sDTTT///r!rctj|d}tj|d}|j\}}|j\}}||ksJtj||fd} tj||fd} |tkrKt t |t |||||t | t | n|tkrKtt |t |||||t | t | nLtt |t |||||||t | t |  | | fS)a Compute the k nearest neighbors of a vector without constructing an index Parameters ---------- xq : array_like Query vectors, shape (nq, d) where the dimension d is that same as xb `dtype` must be float32. xb : array_like Database vectors, shape (nb, d) where dimension d is the same as xq `dtype` must be float32. k : int Number of nearest neighbors. metric : MetricType, optional distance measure to use (either METRIC_L2 or METRIC_INNER_PRODUCT) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rrr) r r r r&r' knn_L2sqrrr)knn_inner_productknn_extra_metrics) r,r-rr.r/r0r1r2r3rrs rknnrMsj2 b 2 2 2B b 2 2 2B HEB XFB 7777 "a(((A "a ***A  RLL(2,, r2q(1++x{{     ' ' ' RLL(2,, r2q(1++x{{     RLL(2,, r2vz1 QKK!   a4Kr!hcc |j\}}|j\}}||ksJtj||fd}tj||fd} |dkrtj} || _|| _tj| | _tj|| _ tj | tj|tj|||dnq|dkrdtj tj|tj|||||tj|tj| nt|| fS)a Compute the k nearest neighbors of a set of vectors without constructing an index. Parameters ---------- xq : array_like Query vectors, shape (nq, d) where d is the number of bits / 8 `dtype` must be uint8. xb : array_like Database vectors, shape (nb, d) where d is the number of bits / 8 `dtype` must be uint8. k : int Number of nearest neighbors. variant : string Function variant to use, either "mc" (counter) or "hc" (heap) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rqrrrrJmc) r r r&r int_maxheap_array_trrrrrhammings_knn_hchammings_knn_mcNotImplementedError) r,r-rvariantr0r1r2r3rrheaps r knn_hammingrsJ2 HEB XFB 7777 "a(((A "a(((A$(**>!$$>!$$  %.$$enR&8&8" q     D  N2  r 2 2BAq N1  u~a00    "! a4Kr!c4eZdZdZdZdZddZd dZdZdS) KmeansaObject that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator c ||_||d|_d|vrt|_nt |_|D]X\}}|dkr"|dks|dkrt}||_-t|j|t|j||Y| dS)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False Fprogressive_dim_stepsgpuTN) r1resetr"ProgressiveDimClusteringParameterscpClusteringParametersitems get_num_gpusgetattrsetattr set_index)rzr1rkwargsvs rr|zKmeans.__init__s  1  "f , ,8::DGG*,,DGLLNN ' 'DAqEzz99R$A###A&&&& r!cl|j}|jjtkre|jjrt ||_nt||_|jr'tj |j|j|_dSdS|jrt|j}nt}||_ dS)N)ngpu)r1r __class__r spherical IndexFlatIPindex IndexFlatL2rr index_cpu_to_all_gpusGpuProgressiveDimIndexFactoryProgressiveDimIndexFactoryfac)rzr1rs rrzKmeans.set_indexs F 7  4 4 4w  ,(^^ (^^ x T"8$(SSS  T Tx 33BBB022DHHHr!Nc\|t||_d|_d|_d|_dS)zg prepare k-means object to perform a new clustering, possibly with another number of centroids N)rgr centroidsobjiteration_stats)rzrs rrz Kmeans.resets2 =VVDF#r!c tj|d}|j\}}||jksJ|jjt krxt||j|j}|>|j\}}||ksJtj | |j | ||j|nZ|J|J|jjrJt!||j|j}| |t#||jtj|j } | |j||_ |j fdt- D tjd D|_d fd D|_|jjdkr |jd nd S) a  Perform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rrNc:g|]}|Sr)at).0r_statss r z Kmeans.train..>s#:::!:::r!cg|] }|j Sr)r)rsts rrz Kmeans.train..?s444RV444r!z,obj time time_search imbalance_factor nsplitc.g|]fdDS)c2i|]}|t|Sr)r)rfieldrs r z+Kmeans.train...Cs% @ @ @5UGB&& @ @ @r!r)rr stat_fieldss @rrz Kmeans.train..Bs?    A @ @ @K @ @ @   r!rrr)r r r r1rrr Clusteringrr copy_array_to_vectorravelrtrainrrProgressiveDimClusteringrrvector_float_to_arrayreshaperrXr:rrsplit) rzrdweightsinit_centroidsrr1clusncr3rrrs @@rrz Kmeans.train s4  ) 4 4 4w1DF{{{{ 7  4 4 4a11D)'-BQwwww*>+?+?+A+A4>RRR JJq$*g . . . .???!)))w( ( ( (+Atvtw??D JJq(1++tx 0 0 0/?? "**46155$::::eEJJLL&9&9:::844e44455DJJLL         $x}q00tx||c9r!cLtj|d}|j Jd|j|j|j|j|d\}}||fS)Nrrzshould train before assigningrJ)r r rrrrsearchr)rzrdrrs rassignz Kmeans.assignHs  ) 4 4 4~))+J)))  t~&&&z  A&&1wwyy!''))##r!r)NN) rrrrr|rrrrrr!rrrsu@. $$$$::::::::x$$$$$r!rc@t|tjjSr) isinstance collectionsabcSequencercs r is_sequencerUs a1 2 22r!c D|j\}}tj|d}t|rtj|d}|j|fksJt |dzdz}tj||fd}t||t|t|t||nQ||zdzdz}tj||fd}t|||t|t|||S)a> Pack a set integers (i, j) where i=0:n and j=0:M into n bitstrings. Output is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. If nbit is an integer: all entries takes nbit bits. If nbit is an array: entry (i, j) takes nbit[j] bits. rqrrI) r r r rrgsumr&pack_bitstrings_cr)rPnbitrM code_sizebs rpack_bitstringsrZs$ 7DAq Qg...A4 K#D888zaT!!!!aA-.. Ha^7 3 3 3 q(4..(1++x{{I G G G GX\a' Ha^7 3 3 3!Qhqkk8A;; JJJ Hr!c $|j\}}|tj|d}t|}t |dzdz}||ksJtj||fd}t||t|t||t|n[|}||zdzdz}||ksJtj||fd}t|||t||t||S)a Unpack a set integers (i, j) where i=0:n and j=0:M from n bitstrings (encoded as uint8s). Input is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. Two forms: - when called with (array, M, nbit): there are M entries of size nbit per row - when called with (array, nbits): element (i, j) is encoded in nbits[j] bits Nrqrrr) r r r rrgrr&unpack_bitstrings_cr)r M_or_nbitsrrrr min_code_sizerPs runpack_bitstringsr us$7LAy |#Jg>>> IITXXZZ!^122 M)))) HaV7 + + + q(4.. QKKHQKK 1 1 1 1 TA!+ M)))) HaV7 + + + q$ Y  = = = Hr!)r6)r6N)rS)Nrr)rr)"numpyr faiss.loaderr collections.abcrrr$r'r5r>rDlrandrGrRrVrUr`rerorirurrrwrrrrrrrrr rrr!rrsD $$'0A2   ,   888 > :    N========@,8$3333l0000pS$S$S$S$S$S$S$S$t333$   2(      r!