Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New attention masking layers #86

Open
7 tasks
soran-ghaderi opened this issue Apr 20, 2023 · 0 comments
Open
7 tasks

New attention masking layers #86

soran-ghaderi opened this issue Apr 20, 2023 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed issue list A list of issues closely related jax Related to JAX numpy Related to Numpy pytorch Related to Pytorch Stale tensorflow Related to Tensorflow

Comments

@soran-ghaderi
Copy link
Member

soran-ghaderi commented Apr 20, 2023 •
edited
Loading

Description: We need to implement several new attention masking layers in our model to improve its performance on specific tasks. The following masking layers need to be implemented:

  • Global
  • Block local
  • Band
  • Dilated #87
  • Random
  • Compound
  • axial

It is important to carefully consider the design and implementation of these masking layers to ensure they are effective and efficient.

Deadline for each layer: 2 weeks after opening the issue. After the deadline, the issue opened will be closed to make it available for other contributors.
@soran-ghaderi soran-ghaderi added enhancement New feature or request help wanted Extra attention is needed good first issue Good for newcomers issue list A list of issues closely related tensorflow Related to Tensorflow pytorch Related to Pytorch jax Related to JAX numpy Related to Numpy labels Apr 20, 2023
@github-actions github-actions bot added the Stale label Jan 17, 2024
@soran-ghaderi soran-ghaderi reopened this Jan 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed issue list A list of issues closely related jax Related to JAX numpy Related to Numpy pytorch Related to Pytorch Stale tensorflow Related to Tensorflow
Projects
@soran-ghaderi TransformerX development workflow
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
1 participant
@soran-ghaderi