Merging tensors of larger models

Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook

It shouldn't be hard to merge tensors with my https://github.com/kir-gadjello/zipslicer library, but it's pure Python! If you want to keep the project pure C++ you might want to write a standalone gist script that uses zipslicer to unpack weight shards into binary files.

Thanks! The bigger problem now is that I am out of disk space, haha!
Anyway, will try to figure out something later

@ggerganov

Leave a tip jar to get a @ggerganov bigger SSD and / or macbook :D

Its kinda pointless now but I was able to merge the 30B and 65B with this core bit of hackery added to the convert script.

+    fname_model = sys.argv[1] + "/consolidated." + str(i).zfill(2) + ".pth"
+    model_i = torch.load(fname_model, map_location="cpu")
+    
+    # Since the models are split, we need to append the tensors changing the shape/size
+    for k, v in model_i.items():
+        if k in model:
+            if model[k].dtype != v.dtype:
+                print("ERROR: Tensor types do not match: ", model[k].dtype, " vs ", v.dtype)
+                sys.exit(1)
+            elif len(model[k].shape) == 1:
+                print("Skipping tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+                continue
+            elif k == "output.weight":
+                print("Concatenating tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+                model[k] = torch.cat((model[k], v), dim=0)
+                print("New shape: ", model[k].shape)                
+                continue
+            elif "tok_embeddings" in k:
+                print("Concatenating tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+                model[k] = torch.cat((model[k], v), dim=1)
+                print("New shape: ", model[k].shape)
+                continue
+            elif "attention.wo" in k:
+                print("Concatenating tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+                model[k] = torch.cat((model[k], v), dim=1)
+                print("New shape: ", model[k].shape)
+                continue
+            elif "feed_forward.w2" in k:
+                print("Concatenating tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+                model[k] = torch.cat((model[k], v), dim=1)
+                print("New shape: ", model[k].shape)
+            else:
+                print("Concatenating tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype, " with shape: ", model[k].shape)
+                model[k] = torch.cat((model[k], v), dim=0)
+                print("New shape: ", model[k].shape)
+        else:
+            print("Adding tensor: " + k + " with shape: ", v.shape, " and type: ", v.dtype)
+            model[k] = v
+    del model_i```

Fixed with 007a8f6

On startup, we go through all the parts and merge them dynamically in the ggml buffers.

Merge pull request ggml-org#1 from MariusCiocanel/MariusCiocanel-curl…

Merge pull request ggml-org#1 from togethercomputer/support_redpajama

Merge pull request ggml-org#1 from sumo43/master

headers fix; add kquants_iter for hipblas and add gfx803 (ggml-org#1)

make benchmark_kv_stream.py compatible with Windows (ggml-org#1)

chore(user-mode): add apply-user-patches.sh (применение ggml-org#1–gg…

moe-stream : include <cmath> for std::isfinite

Merge pull request ggml-org#1 from TaskPuppyNatani/test/k2-pr1-expand…

Merge pull request ggml-org#1 from WestWaters/k2-horizon-msvc-pretoke…

Merge pull request ggml-org#1 from phase-2-hardware-support

strix-halo: fork overlay (docs + findings + model-search tooling)

docs: decision -- GPU decode attention loses to crossings before it s…

llama : add gpt-oss (ggml-org#15091)

common : Fix corrupted memory error on json grammar initialization (g…

sycl : support --split-mode tensor (ggml-org#24152)

vulkan: chunk batched mat-vec at the column counts that are slow on R…

vulkan : only chunk batched mat-vec for the quant types that were mea…

context: do not reset empty graph-result slots when the phase arena a…

Navigation Menu

Repository navigation

Merging tensors of larger models #1

172 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Merging tensors of larger models #1

Description

Activity

ggerganov commented on Mar 10, 2023

theontho commented on Mar 11, 2023

eous commented on Mar 11, 2023

ggerganov commented on Mar 12, 2023

172 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions