Code

initial grok#169

Code

Merged

dan-garvey merged 52 commits intomainnod-ai/shark-ai:mainfrom grokstarnod-ai/shark-ai:grokstarCopy head branch name to clipboardSep 26, 2024

+607-183

Commits

Commits on Sep 5, 2024

• initial grok

  dan-garvey

  committed
  ba87a04

  Copy full SHA for ba87a04

• use name prefix instead of new dataclass

  dan-garvey

  committed
  de9842b

  Copy full SHA for de9842b

Commits on Sep 6, 2024

• some hacks

  dan-garvey

  committed
  b7965b1

  Copy full SHA for b7965b1

• more hack

  dan-garvey

  committed
  6d3d261

  Copy full SHA for 6d3d261

• fix moe-ffn

  dan-garvey

  committed
  b5f535d

  Copy full SHA for b5f535d

Commits on Sep 9, 2024

• Add in some missing grok specific model structure and constants

  KyleHerndon

  committed
  4095db0

  Copy full SHA for 4095db0

Commits on Sep 12, 2024

• Add attn_output_norm layer

  archana-ramalingam

  committed
  e71630a

  Copy full SHA for e71630a

• Update MOE block in decode

  archana-ramalingam

  committed
  5772a3d

  Copy full SHA for 5772a3d

• Some fixes to the grok model

  KyleHerndon

  committed
  3f2914a

  Copy full SHA for 3f2914a

• Merge branch 'main' into grokstar

  archana-ramalingam

  authored
  7c2e133

  Copy full SHA for 7c2e133

• Revert "Merge branch 'main' into grokstar"

  Show description for e1261f5
  archana-ramalingam

  committed
  e1261f5

  Copy full SHA for e1261f5

• Fix merging main changes

  archana-ramalingam

  committed
  a242bde

  Copy full SHA for a242bde

• Update tensor trace names

  archana-ramalingam

  committed
  bb40d12

  Copy full SHA for bb40d12

• Update moe block test

  archana-ramalingam

  committed
  cfa8420

  Copy full SHA for cfa8420

• Update paged attention block with grok changes

  archana-ramalingam

  committed
  325696f

  Copy full SHA for 325696f

• Update paged attention block with grok changes

  archana-ramalingam

  committed
  48fce0c

  Copy full SHA for 48fce0c

• Add use_grok to MOE block

  archana-ramalingam

  committed
  d9e787c

  Copy full SHA for d9e787c

Commits on Sep 13, 2024

• Use use_grok in MOE block

  archana-ramalingam

  committed
  ab084cc

  Copy full SHA for ab084cc

• Change MOE activation from silu to gelu for Grok

  archana-ramalingam

  committed
  29e3603

  Copy full SHA for 29e3603

• Allow router weight norm for all MOEs

  archana-ramalingam

  committed
  0670e1d

  Copy full SHA for 0670e1d

• Update llm_configs to support llama and grok architectures

  archana-ramalingam

  committed
  a4be20b

  Copy full SHA for a4be20b

• Remove comment

  archana-ramalingam

  committed
  3049f87

  Copy full SHA for 3049f87

• Add optional params for Grok

  archana-ramalingam

  committed
  b8240c8

  Copy full SHA for b8240c8

• Add all models supported in sharktank

  archana-ramalingam

  committed
  5bf30e0

  Copy full SHA for 5bf30e0

• Make rope_freq_base mandatory param

  archana-ramalingam

  committed
  d970944

  Copy full SHA for d970944

Commits on Sep 24, 2024

• small refactor/cleanup

  dan-garvey

  committed
  b1fd818

  Copy full SHA for b1fd818

• more cleanup

  dan-garvey

  committed
  85e2f87

  Copy full SHA for 85e2f87

• this shouldn't have been unrebased??

  dan-garvey

  committed
  7ed9a23

  Copy full SHA for 7ed9a23

• fix use_hf args

  dan-garvey

  committed
  3510634

  Copy full SHA for 3510634

• Make use_grok optional in MOE and Attention blocks

  archana-ramalingam

  committed
  a4ff36a

  Copy full SHA for a4ff36a

• Add use_grok to moe_block_test

  archana-ramalingam

  committed
  940db2f

  Copy full SHA for 940db2f

• fix kv cache test

  dan-garvey

  committed
  bb2f5a1

  Copy full SHA for bb2f5a1

• Add PreGatherMoeBlock to import from layers

  archana-ramalingam

  committed
  b6e52eb

  Copy full SHA for b6e52eb

• Add MOE block export for prefill + decode

  archana-ramalingam

  committed
  b790cb5

  Copy full SHA for b790cb5

• Fix architecture variable

  archana-ramalingam

  committed
  19218f3

  Copy full SHA for 19218f3

• Fix imports

  archana-ramalingam

  committed
  7deb42a

  Copy full SHA for 7deb42a

• Fix rope_freq_base

  archana-ramalingam

  committed
  1b6cb6d

  Copy full SHA for 1b6cb6d

• fix flaky test

  dan-garvey

  committed
  43b20c4

  Copy full SHA for 43b20c4

• Merge branch 'main' into grokstar

  archana-ramalingam

  authored
  d938a08

  Copy full SHA for d938a08

• Add short versions for args

  archana-ramalingam

  committed
  6aeeb4f

  Copy full SHA for 6aeeb4f

• Remove use_hf and use_grok options from llama

  archana-ramalingam

  committed
  cac489c

  Copy full SHA for cac489c

• Move create_kv_cache to utils folder

  archana-ramalingam

  committed
  d5c27fe

  Copy full SHA for d5c27fe

• Fix error

  archana-ramalingam

  committed
  10d6c87

  Copy full SHA for 10d6c87

Commits on Sep 25, 2024

• Merge branch 'main' into grokstar

  archana-ramalingam

  authored
  4816c93

  Copy full SHA for 4816c93

• revert addition of dtype arg

  dan-garvey

  committed
  124503f

  Copy full SHA for 124503f

• Merge branch 'main' into grokstar

  dan-garvey

  authored
  46c6eb6

  Copy full SHA for 46c6eb6

• Remove attention_dtype

  archana-ramalingam

  committed
  f3a8fb1

  Copy full SHA for f3a8fb1

• Merge branch 'main' into grokstar

  dan-garvey

  authored
  dcc1e8f

  Copy full SHA for dcc1e8f

• fix missing parenth

  dan-garvey

  committed
  88e38e2

  Copy full SHA for 88e38e2

• correctly rebase T_T

  dan-garvey

  committed
  430045b

  Copy full SHA for 430045b

Commits on Sep 26, 2024

• nonstrict

  dan-garvey

  committed
  f0a3e31

  Copy full SHA for f0a3e31

• Merge branch 'main' into grokstar

  dan-garvey

  authored
  e5dc9e9

  Copy full SHA for e5dc9e9